Overview

Diffhold is change control for AI agents on Kubernetes. It decides, for each tool call an agent makes, whether that exact change may happen now, and makes sure the credential it hands out can do nothing else.

Diffhold is pre-alpha. The core decision path works end to end and runs in CI on kind, but it hasn't been released, tested on managed clouds or externally reviewed. See Security model for what is still open.

What it does for one tool call

Map. The MCP tool call is turned into Kubernetes objects. Tools that can't be mapped, such as a raw shell, are denied.
Check. The agent's AgentPolicy is evaluated. A denial stops the call unless the policy is in audit mode.
Preview. Writes run a server-side dry-run as the agent's ServiceAccount, and Diffhold computes a field-level diff. If the dry-run fails (invalid manifest, missing RBAC, another admission controller says no), the call is denied.
Hold. If a matching rule needs approval, the call waits. The agent and the approvers see the diff. An approval covers only that diff and is used once.
Grant. Diffhold mints a short-lived token for the agent's ServiceAccount, stores a signed grant describing the change, writes an audit record and lets the call through with the token.

The MCP server then calls the API server with that token. Diffhold's admission webhook compares each request made with an agent token against its grant, so the token can't repeat the change, change a different field or touch a different object.

Components

Component	What it does
ExtMcp policy server	gRPC processor that agentgateway calls for every MCP request (port 9000, mutual TLS).
Dry-run and diff	Server-side apply, scale and delete dry-runs as the agent, with server noise removed from the diff.
Approvals API	Lists pending calls with their diffs and records approvals from authenticated people (port 9001).
Credential broker	Per-call TokenRequest tokens bound to a grant Secret; deleting the Secret revokes the token.
Admission backstop	Validating webhook that only applies to ServiceAccounts in agent namespaces. It fails closed for agents and never affects people or controllers.
Policy controller	Watches `AgentPolicy` objects, reloads them without a restart and writes a `Ready` condition.
Audit log	Hash-chained JSON lines with the decision, the diff, the approvers and the token ID. `diffhold audit verify` checks the chain.
AuthZEN endpoint	Decision-only API for gateways other than agentgateway. It issues no token.

High availability

The chart runs two replicas behind one Service and one webhook. Grants live in the grant Secrets themselves and approvals in ConfigMaps, so any replica can serve any call. A Lease elects one leader to revoke expired grants, expire old approvals and write policy status. If the leader stops cleanly, another replica takes over in about 3 seconds; after a crash it takes up to the 15-second lease.

Requirements

Kubernetes 1.30 or later (token credential IDs and webhook matchConditions).
agentgateway 1.7 or later, with JWT authentication for agents.
An MCP server that takes a per-request Kubernetes token in a header. kubernetes-mcp-server is the tested one.