CORE DIRECTORY // SYSTEM.USER.DIANA_ISMAIL

Labs by Diana — Experiments that ship.

Side projects that got out of hand. AI tools built for problems I kept tripping over — now live, now yours.

Agentic Infrastructure Notesactive

The Control Plane Problem - Why Agent Governance Is the Real Bottleneck

ARTICLE_029

AGENTIC_INFRASTRUCTURE_NOTES // 7_OF_8

PUBLISHED

2026.06.30

READ

~11 MIN

This piece names the three-layer control plane that a production multi-agent system requires: observability (what the agent is actually doing), enforcement (what it is permitted to do), and recovery (what happens when it violates a constraint). These layers are not theoretical - they are the architecture that 12 agents running across roughly 10 repos have forced into existence over 18 months, mostly through failure rather than design.

The industry has started arriving at the same structure independently. Salt Code, ASSERT from Microsoft Build, and the Agent Control Specification each address a slice of the same problem from different angles. None of them is governance in full. Together they confirm that the category is real, the need is urgent, and the solved version does not yet exist.

The_Gap_Nobody_Talks_About_Until_It_Costs_Something

Here is a question worth asking your team right now: if one of your agents does something wrong - calls an API it should not have, produces output that violates a brand constraint, runs a task against the wrong dataset - what happens?

Not "what would we do." What actually happens, in the system, automatically, before a person notices?

Most teams building with agents cannot answer this. They have built the capability layer: agents that can write code, call tools, produce content, orchestrate other agents. They have not built the control layer: the part of the system that monitors what those agents are doing in real time, enforces what they are permitted to do, and responds when a constraint is violated. The agents work. The system is not under control.

This is the governance gap. It is not a security gap, though security is part of it. It is not an alignment gap in the philosophical sense. It is an operational gap: the distance between "we built an agent" and "we can run it reliably, continuously, in production." That distance turns out to be where most of the actual engineering lives.

I built into this gap before I understood it as a category. What I understand now, 18 months and 12 agents later, is that the gap has a shape. It breaks into three layers, each with distinct failure modes, each requiring different instrumentation to close.

The_Three_Layers:_Observability,_Enforcement,_Recovery

Observability is what you can see. Not what you think the agent is doing - what it is actually doing, at the instruction level, in the session where it is running. For a single agent on a single task, this is almost trivial: you watch the output. For 12 agents running in parallel across 10 repos, observability becomes a genuine engineering problem. Which agent touched which file? What was the state of the system when Nix committed that branch? Did Sable's database migration run against staging or against production?

The observability layer I run is EXP_004: a nightly audit across all repos that checks for stale manifests, failing CI, open Dependabot PRs, uncommitted worktree state, and branch hygiene violations. It does not give me real-time visibility into what every agent is doing at every moment. What it gives me is a signal layer that catches drift before it compounds. Observability built reactively is incident response. Observability built before the incidents is governance. I built EXP_004 after the model tiering inheritance bug - not as a direct response to it, but because that incident made visible how long a problem could run silently in a system I thought I was watching.

Enforcement is what the system will not let the agent do, regardless of what the agent has been instructed to do. This is the layer that most governance conversations skip, because it is the hardest one to build. Enforcement requires the control plane to sit between the agent and its action surface - intercepting the action before it executes, evaluating it against a policy, and blocking or routing to review if the policy is violated.

In my system, enforcement is partial and inconsistent. Trust tiers are the nearest thing to an enforcement layer: Autonomous agents run without review; Always-Review agents route every output to a human gate before execution; Conditional agents route based on blast radius. The tier is assigned when the agent is onboarded and updated as track record develops. But trust tiers are a classification, not an enforcement layer. They tell the orchestrator what to do with an agent's output. They do not sit between the agent and the API call at runtime. When a content agent in the W26 batch fabricated first-person anecdotes and they passed through Reid's voice-check - which reviews register and vocabulary, not factual accuracy - there was no layer in the system that caught it before it went live. The enforcement gap was not in the trust tier design. It was in the fact that the policy ("verify that a requested anecdote was supplied, rather than fabricated") was not written into the execution path.

Recovery is what the system does when a constraint is violated. This is the layer most teams think least about until they need it urgently. Recovery is not just rollback - it is the full set of responses available to the system when something has gone wrong: pause the agent, escalate to review, revert the action, trigger a notification, log the incident for post-mortem. Recovery without observability is blind; recovery without enforcement means you are always responding after the fact.

My recovery layer is almost entirely human-in-the-loop. When something goes wrong, I notice it, diagnose it, and fix it manually. The system does not pause itself. It does not escalate automatically. It does not have a recovery protocol that runs without my involvement. That is not a design choice - it is a gap. It means that recovery speed is bounded by when I happen to be watching.

What_the_Industry_Is_Converging_On

Salt Code, announced in June 2026, is a security policy enforcement layer applied inside AI coding assistants across code discovery, creation, review, CI/CD pipeline, and runtime. It is security tooling - its primary framing is code vulnerability and secrets exposure. But what it demonstrates is the pattern: a control layer inserted into the agent's workflow, at multiple lifecycle stages, evaluating outputs against policy before they execute. The layer is not advisory. It is blocking.

ASSERT, introduced at Microsoft Build 2026, is policy-driven evaluation tooling that converts organisational requirements into test scenarios for agent behaviour - defining what a compliant output looks like, then flagging deviations across sessions. The insight behind ASSERT is that agent loops are not stateless functions - they are stateful processes that can drift across sessions. Regression testing catches the drift before it becomes a production failure.

The Agent Control Specification - also from Microsoft Build - addresses where in an agent loop the control logic should sit. Not as a post-hoc audit layer, not as a rule in a system prompt, but as a defined interface point in the loop itself: before tool calls, after outputs, at handoffs between agents. The specification makes governance a first-class architectural concern rather than something bolted on after the capability layer is built.

MCP sits adjacent to all of this. MCP defines the interface surface for tool calls - the standardised protocol through which agents call external tools, APIs, and data sources. Agent Control Specification defines the policy logic that governs which tool calls are permitted, under what conditions, with what audit trail. Together they form the interoperability and policy stack: MCP gives the agent a consistent way to reach its action surface; ACS governs what it can do once it gets there. An agent that calls tools via MCP and operates under an ACS policy layer has a defined, auditable, controllable action surface. Most agents running today have neither.

What these three projects confirm is that the industry has identified the same problem from three independent directions. The pattern is not coincidence. A production-grade agentic deployment requires a control plane. The demos do not have one. The difference between the demo and the deployment is whether the control plane exists.

Two_Failures_That_a_Control_Plane_Would_Have_Caught_Earlier

The model tiering inheritance bug. For several weeks, agents were inheriting the Opus model tier from the parent session when spawned as subagents, rather than running on the tier specified in their definitions. Nothing broke. Everything worked - it just worked on Opus, at Opus cost, with Opus latency. The observability signal was in the usage data, but I was not watching that signal closely enough to catch drift at the subagent level. An observability layer watching model tier at spawn time, with an alert on tier mismatch, would have surfaced this within one session rather than across weeks. The cost was not catastrophic. The principle is: silent failures in multi-agent systems accumulate in ways that single-agent failures do not. You need the signal layer before you need it.

The fabricated-anecdote incident. A batch of drafted content pieces carried first-person anecdotes that were fabricated. They went through Reid's voice-check - which flagged for register and vocabulary, not for factual verification of whether the anecdote was supplied in the brief. There was no dedicated fact-verification step in the path at all. The enforcement gap was not in the individual reviewing step; it was in the fact that verification of whether a requested element (a lived encounter) had actually been supplied, rather than invented by the agent, was not a defined policy in the execution path. A fact-grounding check - "was this anecdote supplied in the brief, or did the agent fabricate it?" - would have caught this before the batch went to publication. Policies that live only in documents do not run.

What_Remains_Unsolved

I want to be precise about where this lands, because there is a temptation to read the above as a solved problem. It is not.

What I have is an observability layer (EXP_004, nightly, ten repos), a classification-based enforcement layer (trust tiers, which are planning-time policy, not runtime policy), and a recovery layer that is almost entirely manual. The control plane is partial, inconsistent, and human-dependent in all the places that matter most.

Salt Code, ASSERT, and Agent Control Specification each solve a slice of this. None of them - and no combination of them currently available - solves runtime governance at the scale of 12 agents running continuously across roughly 10 repos. The observability problem across simultaneous sessions is unsolved. The enforcement problem at the tool-call level is not productised in a way that integrates cleanly with how I have built the system. The recovery problem is largely unaddressed by the industry at the orchestration layer.

This is piece seven of a series about the plumbing layer of agentic AI. The observability piece is coming next, and it goes deeper into this specific gap: what you cannot see in a running multi-agent system, why the signal architecture matters more than the monitoring tooling, and what drift looks like before it looks like a failure.

The governance problem is real. The control plane is necessary. The solved version does not yet exist.

Agent GovernanceControl PlaneAgentic AIProduction SystemsMulti-Agent

KEY_TAKEAWAYS

TAKEAWAY_01

The governance gap is not a security problem - it is an operational one. The distance between a working agent and a governable agent is the three-layer control plane: observability (what the agent is actually doing), enforcement (what it is permitted to do), and recovery (what happens when it violates a constraint). Most production agentic systems have fragments of each layer. Very few have all three.

TAKEAWAY_02

Policies written in documents are not enforcement layers. The W26 fabricated-anecdote incident passed through Reid's voice-check because verification of whether a requested anecdote was supplied in the brief, versus fabricated by the agent, was not a policy in the execution path - it was an implicit assumption that "lead with a lived encounter" could only succeed if the encounter existed. Salt Code's insight, and Microsoft Build's Agent Control Specification, is that control logic has to sit inside the agent loop at defined interface points, not in documentation the agent was told to follow. The gap between "we have a policy" and "we have an enforcement layer" is where most governance failures live.

TAKEAWAY_03

Runtime governance at scale is the unsolved problem. Salt Code, ASSERT, and Agent Control Specification each address a slice of the control plane. In my assessment, none of them - and no current combination of them - solves the full governance problem for 12 agents running continuously across roughly 10 repos simultaneously. The industry has converged on naming the problem correctly. The tooling to solve it at real operational scale is still being built.

SYSTEM.INT // 2026 LABS_CORE v2.78.2

LATENCY: STATUS: NOMINAL