Diffhold pre-alpha

Every change an AI agent makes to your cluster, held until it’s checked.

Diffhold previews each agent change with a server-side dry-run, checks it against policy, waits for a named person when the change is risky, and then lets the call through with a credential that can make only that change, once.

Kubernetes 1.30+ · agentgateway 1.7+ · any MCP agent · Apache-2.0

call 6663a7ce3960agent sre-bottool resources_delete
{"kind": "ConfigMap", "namespace": "team-a", "name": "flags"}
- ConfigMap team-a/flags (deleted)
  1. MAPPED1 object · delete
  2. POLICYrule destructive needs approval
  3. DRY-RUNas agents/agent-sre-bot · 11 ms
  4. HELDfor group:team-a-oncall · expires in 15m
  5. APPROVEDby alice · same diff as shown
  6. TOKENsingle use · 30s · bound to this delete
  7. REPLAYsame token again → denied by backstop
The gap

Agents run kubectl now. Your controls still think in tool names and roles.

In April 2026 a coding agent with a broad token deleted a production database and its backups in nine seconds. Nothing asked what the change would do before it ran.

Tool names are too coarse

An MCP gateway can allow or deny resources_create_or_update. It can’t tell scaling to 3 from scaling to 300, or a new image from a privileged container.

RBAC grants windows

Kubernetes RBAC and access brokers grant a role for a period of time. Nothing approves the one change a person actually looked at.

Tokens outlive the decision

Whatever credential the agent holds works for everything its ServiceAccount allows, for as long as it lives, including after it leaks.

How it works

Five checks on every tool call, in this order

Diffhold plugs into agentgateway as an ExtMcp policy processor. In testing on kind, a write’s dry-run added about 11 ms (p50).

Map

The tool call becomes Kubernetes objects: kind, namespace, name, body. Tools that can’t be mapped, like raw shell, are denied.

Check

The agent’s AgentPolicy decides by verb, kind and namespace, plus CEL conditions on the object.

Preview

A server-side dry-run runs as the agent’s ServiceAccount and returns a field-level diff.

Hold

Risky rules wait for named approvers. An approval covers only the diff that was shown, and only once.

Grant

The call goes through with a per-call token. An admission webhook rejects anything the token tries beyond that change.

Per-call credentials

A leaked agent token can make one change, once.

Attempt with the token from “scale web to 4”Result
Scale team-a/web to 4ALLOW
Same change againDENY
Scale to 5 insteadDENY
Delete the DeploymentDENY
Read a SecretDENY
Apply a different spec to an approved objectDENY
Use a grant edited from scale to deleteDENY
A ServiceAccount token minted outside DiffholdDENY

From the end-to-end suite that runs on kind in CI on every push.

Diffhold mints a short-lived ServiceAccount token for each allowed call and records a grant for it: the verb, the objects, the approved replica count or a hash of the exact object the dry-run predicted.

A validating admission webhook, scoped to agent ServiceAccounts only, compares every request against that grant. Humans and controllers never touch it, and if Diffhold is down, agent writes fail closed.

Grants are signed with a key agents can’t read, used once, and revoked when they expire.

How the credential path is secured →
AgentPolicy

One Kubernetes resource per agent

Policies live in the cluster and report their status like any other resource. The v1alpha1 schema is frozen: later releases only add optional fields.

apiVersion: diffhold.dev/v1alpha1
kind: AgentPolicy
metadata: {name: sre-bot, namespace: diffhold-system}
spec:
  subjects: [{agent: sre-bot}]       # JWT sub from the gateway
  credentials:
    serviceAccount: agents/agent-sre-bot
    grantTTL: 30s
  limits: {maxObjectsPerCall: 3, ratePerMinute: 120}
  rules:
  - name: operate-team-a
    verbs: [apply, scale]
    resources: [{kinds: [Deployment], namespaces: [team-a]}]
    conditions:
    - "object.spec.replicas <= 5"
  - name: destructive
    verbs: [delete, exec]
    resources: [{kinds: ["*"], namespaces: [team-a]}]
    approval: {approvers: [group:team-a-oncall], timeout: 15m}

Rules match verbs, kinds and namespaces in order. CEL conditions see the proposed object, so a limit can read replicas or every container image.

Approvals name users or groups as your API server knows them. Approvers sign in with their own Kubernetes or OIDC token; the agent can never approve itself.

Audit mode records what would have been denied without blocking, so you can roll a policy out safely.

Policy tests run in CI with diffhold test, which checks expected allow, deny and approval for each tool call.

AgentPolicy reference →
Where it fits

Works alongside the tools you already run

Teleportdecides who can reach the cluster
Kyvernodecides what may be deployed
Diffholddecides what change an agent may make, right now
Security

Threat-modelled before anyone runs it

The threat model lists 25 threats with status and evidence. The seven that blocked any outside use are fixed, and each fix started with a test that reproduced the attack.

IDENTITY
The agent identity comes only from the gateway’s authenticated metadata, such as the JWT subject. Headers are ignored.
TRANSPORT
The gateway connection requires mutual TLS. The Helm chart generates both certificates.
CONTENT
Apply tokens are bound to a hash of the exact object the dry-run predicted.
GRANTS
Grants are HMAC-signed with a key only Diffhold can read, and used once.
APPROVERS
Approvers authenticate as themselves; counts, groups and timeouts are enforced and recorded.
EVIDENCE
Hash-chained audit log with diffs and approvers, and signed checkpoints of its head. Secret values are always redacted.
SELF-CHECK
Diffhold checks every agent ServiceAccount for permissions that could switch it off, and can refuse to serve those agents.
LIMITS
Per-agent rate limits and a cap on open approval requests keep a runaway agent from flooding people.

Security model and open items →

Status

Pre-alpha. The core decision path works end to end.

DONEFoundation

  • ExtMcp plugin and AuthZEN endpoint
  • Dry-run and diff, approvals, per-call tokens, backstop
  • CRD, controller, HA with leader election
  • Helm chart, multi-arch image, e2e on kind in CI
  • All 7 P0 threat-model fixes, API frozen
  • P1 hardening: permission self-check, signed audit chain, admin TLS, approval caps

NEXTOSS v0.1

  • Slack approvals tested on a real workspace
  • Helm tools for agents (v0.2)
  • Load tests on a cloud cluster
  • Cloud installs on EKS, GKE, AKS, RKE2
  • External review of the credential path

NOT YETPublic

  • The source and image are private until the v0.1 release
  • Not tested on EKS, GKE, AKS or RKE2 yet
  • Not yet run with a real LLM agent; tests use a scripted client
  • No chart repository; install from a checkout