Expose delegate client JWKS and token-exchange policy in the CRD by jhrozek · Pull Request #7019 · stacklok/toolhive

jhrozek · 2026-11-02T15:08:50Z

Summary

Operators can only configure a pre-provisioned delegate client for RFC 8693 token exchange through the RunConfig today. The operator CRD cannot express the key-based (private_key_jwt) form that the auth server supports, so a broker that rotates its key has to edit the RunConfig by hand. This PR exposes the delegate client settings on the operator API and wires them through to the auth server.

What changed:

The delegate client on MCPExternalAuthConfig and VirtualMCPServer gains tokenEndpointAuthMethod, jwksRef (a ConfigMap key holding the public JWKS), tokenEndpointAuthSigningAlg, grantTypes, redirectUris, and a tokenExchange policy block. Admission rules mirror the runtime validator for the cases CEL can express.
The operator mounts each jwksRef read-only and points the RunConfig jwks_file at it.
The pod rolls when a referenced JWKS ConfigMap changes. The JWKS bytes are part of the pod-template checksum, so the next reconcile restarts the auth server with the new keys. Without this, a key rotation would never reach a running pod.
A VirtualMCPServer whose delegate clients are configured only on the canonical inboundGrants.tokenExchange list is now accepted. The runtime validator previously counted only the deprecated top-level list, so such a server was rejected at reconcile.
The trust-mode error for a private_key_jwt delegate without a token_exchange block now names the one-line change that keeps the old behaviour.
Architecture docs for the auth server and the operator describe the CRD fields and the rollout behaviour.

Type of change

Test plan

Unit tests (task test): run, exit 0, no failure output. The run includes the new validator test for canonical delegate clients.
E2E tests: focused Ginkgo run of the new kind suite VirtualMCPServer delegate client jwksRef on a kind cluster built from this branch. 4 of 4 specs passed: the read-only mount with a matching-key exchange, a wrong key under a registered kid, a foreign subject token refused under restricted, and a two-stage key rotation with a pod roll.
Linting (task lint-fix): not run yet.
Manual testing: none beyond the kind suite.

The envtest CEL suite in cmd/thv-operator/test-integration/mcp-external-auth/delegate_client_cel_test.go has been written but not run. It is the check for the admission rules and should be run before merge.

API Compatibility

This PR does not break the v1beta1 API. Every new field is optional. The only other change is that the deprecated top-level delegateClients now rejects the new fields. Those fields do not exist on main, so no stored object is affected.

Does this introduce a user-facing change?

Yes. Before: the operator CRD could not configure a private_key_jwt delegate client, and a VirtualMCPServer that configured delegates only on inboundGrants.tokenExchange was rejected at reconcile. After: both work. Specifically:

New delegate client fields on MCPExternalAuthConfig and VirtualMCPServer, documented in the CRD reference.
Editing a referenced JWKS ConfigMap rolls the auth server pod. Editing it does not hot-reload the keys.
The deprecated top-level delegateClients rejects the new fields and points users at inboundGrants.tokenExchange.delegateClients.
Redirect URIs must be https with no fragment. A loopback redirect URI is refused when the operator reconciles, and the resource reports it in its status conditions. The API server does not refuse it at admission, because the operator has no admission webhook.

Special notes for reviewers

The trust-mode requirement for private_key_jwt delegates is already on main (Require a trust mode for private_key_jwt delegates #7004). This PR only changes the error text.
Loopback redirect URIs are refused at reconcile, not at admission. CEL cannot express loopback reliably, and there is no webhook. Status reporting is the accepted behaviour, following the existing forceConfidentialRedirectUris precedent.
The JWKS reference uses a new local ConfigMapKeyRef type rather than corev1.ConfigMapKeySelector, so optional cannot be set and both name and key are required.
The generated CRD YAML is most of the diff. It appears once per copy under deploy/charts/operator-crds/files and templates, across both MCPExternalAuthConfig and VirtualMCPServer.
Exchange refresh tokens are not part of this PR. They follow in a separate PR with their own end-to-end coverage.
The commit history is one logical change per commit, with the comment and migration-hint commits riding along.

The comment said a key implies its signing algorithm. An RSA key without alg does not imply RS256; RS256 is only the default when the keys do not say, because RS and PS are both possible. Say what the function returns. Co-authored-by: Mecatl <noreply@mecatl.dev>

A private_key_jwt delegate without a token_exchange block is refused since the trust-mode rule landed. Users upgrading from an earlier release may have such a client that relied on the implicit any behaviour, so the error now names the one-line change that restores it. Co-authored-by: Mecatl <noreply@mecatl.dev>

A broker that keeps one client_id across key rotation needs the pre-provisioned client fields on the operator API, not only in the RunConfig. Delegate clients on inboundGrants.tokenExchange.delegateClients now accept tokenEndpointAuthMethod, jwksRef, tokenEndpointAuthSigningAlg, grantTypes, redirectUris and a tokenExchange source policy. jwksRef is a ConfigMap key holding the public JWKS. The operator mounts it read-only under its own base path and sets jwks_file to match. Editing the ConfigMap does not restart the pod yet. Admission mirrors the runtime validator so the CRD cannot produce a config the auth server refuses. clientSecretRef is now optional in the schema: CEL requires it unless the method is private_key_jwt, where it and jwksRef swap places, and jwksRef.optional is rejected. grantTypes, when set, must include token exchange; redirectUris and refresh_token need authorization_code; tokenEndpointAuthSigningAlg needs private_key_jwt and a supported algorithm. A client that sets grantTypes or uses private_key_jwt must declare tokenExchange.selfIssuedSubjectTokens, any or restricted. Every delegate client, private_key_jwt included, still counts as confidential for the cleartext-transport rule. The deprecated top-level delegateClients shares the type, so the new fields, the CEL rules and the JWKS mount apply to it as well. effectiveDelegateClients picks the canonical list or falls back to the deprecated one, so a jwksRef there never leaves jwks_file pointing at a missing file. Co-authored-by: Mecatl <noreply@mecatl.dev>

A delegate client's jwksRef ConfigMap is mounted with subPath, so a running pod never sees an edit. Without a pod-template change and a reconcile on the edit, rotating the broker's key would need a manual restart. The embedded-auth ConfigMap checksum now also covers the JWKS bytes of the delegate clients the runtime config is built from. The section is hashed only when a jwksRef exists, behind its own marker, so the checksum of every config without one is byte-identical and no existing pod rolls on upgrade. A missing ConfigMap or key is reported as a terminal InvalidCABundle error through ValidateEmbeddedAuthServerCABundles, as CA bundles are. The ConfigMap indexes for MCPExternalAuthConfig and VirtualMCPServer include the JWKS ConfigMap names, so editing one enqueues the dependent reconciles. Co-authored-by: Mecatl <noreply@mecatl.dev>

The operator now rolls the auth server pod when a delegate client's JWKS ConfigMap is edited, but the architecture docs still said an edit needs a manual restart. Say what happens on an operator-managed edit, and keep the restart instruction for the RunConfig path, where the runner still reads jwks_file once at startup. Co-authored-by: Mecatl <noreply@mecatl.dev>

The delegate client JWKS feature depends on behaviour a fake API server cannot show: the auth server pod reading the mounted key at the path the operator sets, and a pod roll when the JWKS ConfigMap changes. This adds a kind suite on a VirtualMCPServer with a restricted private_key_jwt delegate that covers the mount and a matching-key exchange, a wrong key under a registered kid, a foreign subject token that restricted refuses, and a two-stage key rotation that waits for the roll before it checks which key is trusted. A retired kid is no longer registered, so the server refuses it as a malformed request (400 invalid_request), not as a wrong key under a registered kid (401 invalid_client). Co-authored-by: Mecatl <noreply@mecatl.dev>

codecov · 2026-11-02T15:17:47Z

Codecov Report

❌ Patch coverage is 95.23810% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.48%. Comparing base (a4aef63) to head (38a435b).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/thv-operator/pkg/controllerutil/authserver.go	95.55%	4 Missing ⚠️
...or/controllers/virtualmcpserver_authz_configmap.go	85.00%	3 Missing ⚠️
...perator/api/v1beta1/mcpexternalauthconfig_types.go	98.07%	1 Missing ⚠️
...perator/controllers/virtualmcpserver_deployment.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7019      +/-   ##
==========================================
- Coverage   80.50%   80.48%   -0.02%     
==========================================
  Files         852      852              
  Lines       87035    87170     +135     
==========================================
+ Hits        70065    70160      +95     
- Misses      16965    17005      +40     
  Partials        5        5

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

CI lint failed on three findings in the delegate JWKS change: a field alignment that gci rejects in the deployment test, and two functions over the cyclomatic complexity limit. Split the VirtualMCPServer ConfigMap indexer into an authz helper and an auth server helper, and move the spec-level auth server checks into validateAuthServerSpec so the reconciler has one branch for them. The checks and their messages are unchanged. The kind test's exchange helper always received the same client ID, so it now uses a suite constant and the parameter is gone. Co-authored-by: Mecatl <noreply@mecatl.dev>

Lint still flagged two helpers in the delegate JWKS kind suite whose client ID argument always received the suite's delegate client. Both now use the suite constant, so the parameter is gone from each signature and its call sites. Co-authored-by: Mecatl <noreply@mecatl.dev>

jhrozek · 2026-11-02T21:30:46Z

Please hold off on reviewing this one for now.

This PR exposes tokenExchange.apiAudience in the CRD, with a rule that it must be one of the client's audiences and a field doc saying the client may request it even when the subject token doesn't cover it. That's the behaviour the fix for "Configured delegate client with api_audience can exchange for API access tokens" (#7026) removes.

The fix lands first. After that I'll rework this PR to the new shape before it merges: subjectTokenAudiences as an input-only list (only valid with selfIssuedSubjectTokens: restricted), and audiences as outputs only. The CRD field hasn't been released, so there's nothing to migrate. I'll leave another note when it's ready for review again.

jhrozek and others added 6 commits November 2, 2026 10:59

jhrozek requested review from ChrisJBurns, JAORMX, amirejaz, blkt, jerm-dro, rdimitrov, reyortiz3 and tgrunnagle as code owners November 2, 2026 15:08

github-actions Bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Nov 2, 2026

jhrozek mentioned this pull request Nov 2, 2026

Expose exchange refresh token issuance in the CRD #7021

Open

3 of 11 tasks

This was referenced Nov 2, 2026

Authorization server: refuse every failed private_key_jwt client assertion with 401 invalid_client #7028

Open

Phase 1, Track: ToolHive — The authorization server and gateway changes the phase 1 flow needs stacklok/mecatl#2543

Open

Navigation Menu

Repository navigation

Expose delegate client JWKS and token-exchange policy in the CRD - #7019

Expose delegate client JWKS and token-exchange policy in the CRD#7019
jhrozek wants to merge 8 commits into
mainfrom
stable-client-v2-f-3

jhrozek commented Nov 2, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

jhrozek commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jhrozek commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of change

Test plan

API Compatibility

Does this introduce a user-facing change?

Special notes for reviewers

Uh oh!

codecov Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jhrozek commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jhrozek commented Nov 2, 2026 •

edited

Loading

codecov Bot commented Nov 2, 2026 •

edited

Loading