The_Seven_Control_Points

An agent posts to a public account with no audit trail. A twin services user conversations with zero spend visibility. A pipeline runs in production with no way to stop it without a code deploy.

These are not theoretical failure modes. They are the categories of production failure I found when I ran an audit template across my own fleet of live agents.

Most teams building production AI systems audit for one or two failure categories. Kill switch, maybe. Spend if they are risk-conscious. They ship anyway. The other five become production surprises.

I built a template that asks all seven questions. I ran it against four agents already live at production scale. Three came back RED. This is the framework that found them, the gaps that surfaced, and what closing the gaps actually required.

Production agents need answers to seven questions. Not ideally. Not in a mature future state. Before they go live. Here is what each control point asks.

Runtime is the execution environment: where the process lives, how it starts, how it restarts on failure, and what its lifecycle boundaries are. Can you name the hosting platform, the start mechanism, the restart policy, and the health check that tells you the agent is still running? An agent without a documented restart policy is an agent that goes dark when it fails. You do not know it has failed until a user tells you.

Identity covers who is allowed to invoke the agent, how that identity is verified, and what the agent can do on behalf of that identity. Are inbound requests authenticated? Is there an authorisation model that bounds what the agent can touch? How does the agent's own outbound identity work - where do its API keys live, how do they rotate? An agent with no authentication is a public LLM call endpoint. Anyone with the URL can drive costs and trigger actions.

Data is what the agent reads and writes. Can you enumerate every data source the agent accesses? Every table or resource it can mutate? What sensitivity classification applies to the data it handles? Is there isolation between users or accounts? An agent that can read cross-user data or write without scoping is an agent that will eventually expose something to someone who should not see it.

Tools are the external capabilities the agent invokes: APIs, LLM providers, shell commands, integrations. Can you list every external service the agent can call? Which LLM provider and model does it use? Are you versioning the prompts that drive the LLM, or are they floating? An agent that can call any API is an agent with an unknown blast radius. An agent calling an unversioned prompt is an agent that might behave differently tomorrow for reasons you did not control.

Spend is what the agent costs to run. Can you measure the cost per call? Can you see aggregate spend? Is there a provider-level cap configured on your LLM account that bounds the worst-case runaway cost? Is there an in-app spend cap or alert? Spend control is two-layered. The outer wall is the provider-level cap configured on every LLM provider account - this bounds fleet-wide cost runaway at the account level, before any agent-level logic runs. The inner wall is in-app controls: spend visibility, usage alerts, per-call cost tracking. A Spend RED verdict on in-app controls does not mean existential cost risk if the outer wall is in place - it means the operational visibility layer is missing. Both walls need answering.

Observability is whether a human can understand what the agent is doing, has done, and why. Are logs structured? Are errors captured and surfaced? Is there an audit trail of significant actions? Where do logs live and how long do they persist? Is there alerting when things fail? An observable agent can be debugged. An unobservable one fails silently and you only discover it when a user complains.

Kill switch is whether you can stop the agent without a code deploy. Is there a runtime-mutable stop mechanism - a feature flag in a database, an environment variable toggle, an admin control? Can you access it without engineering work? If the agent is scheduled or autonomous, does the stop mechanism survive a scheduled trigger, or do you have to wait for the next scheduled run to abort? An unkillable agent in production is the highest-risk structural gap. It can accrue unbounded costs, publish unreviewed content, mutate data without a stop signal, or exhaust a third-party quota with no emergency brake.

Why_This_Template,_Why_Now

As Geoverity, ig-autopilot, telegram-digital-twin, and FitChecker simultaneously moved into agent-shaped features, it became clear that audit practice had to be standard - not ad hoc per project, not built from a checklist that changes between agents.

I did not design this template theoretically and then test it. I built it because I needed to audit fast and name gaps clearly across four active agents. I ran it against my own fleet. Three of them came back RED.

Applied:_What_the_Audit_Found

Four agents audited. Three overall verdicts.

GEOAudit came back YELLOW. The Spend control point surfaced a named gap: no operator-side LLM spend alert. If a prompt goes wrong or a retry storm starts, the LLM provider account can be exhausted with no in-system signal before the provider cuts off. The credit system caps user-driven spend, but not the underlying API cost. Observability came back YELLOW too - logs are on Railway with no external aggregator, no search window beyond seven days, no alerting for application-level errors. The Kill switch verdict initially came back YELLOW and was closed to GREEN during the same audit session by a global_halt feature flag that gates all new runs through a single chokepoint check. Two YELLOWs overall; no REDs.

ig-autopilot initially came back RED. The Observability control point failed the verdict. The pipeline had no structured logging, no audit trail of what was published, no persistent record of success or failure. Once the post went live to Instagram, there was no in-system way to know what was published, from which template, or whether the run succeeded. If the scheduler stub were ever implemented and automation enabled, silent failures became undetectable. A follow-up PR closed the audit trail gap within the same audit period - Observability is now YELLOW and the overall verdict for ig-autopilot is YELLOW. The RED was real at audit time. Naming it is what unblocked the fix.

FitChecker came back YELLOW. The LLM integration is substantially mature - instrumentation via agent-sdk, cost tracking, retry policy, abort propagation. The verdicts landed on the cautious side: Spend YELLOW because in-app spend alerts could be more granular. Kill switch YELLOW because there is no emergency stop without taking the service offline. No REDs.

The labs canonical twin - the public-facing AI twin at labs.dianaismail.me - came back RED on three control points.

First: Spend is RED. Token usage is logged to stdout only. There is no cost counter, no budget alert, no way to know what the twin costs per day without checking the OpenAI billing dashboard manually. A coordinated multi-session use pattern could accrue significant cost with no in-system signal.

Second: Observability is RED. All logging is unstructured console calls. No structured JSON logger. No correlation IDs on requests. No audit trail of what was sent to OpenAI or from which session. In a serverless environment, Railway log retention is the only window - seven days. After that, the evidence is gone.

Third: Kill switch is RED. There is no mechanism to stop the twin's LLM calls without deploying a code change or pausing the entire Railway service. No feature flag. No maintenance mode. No admin toggle. The twin is public-facing with no authentication on the web interface - any user who reaches the chat URL can drive LLM calls.

These are not design oversights. They are gaps that the audit framework caught, that the day-to-day operational model had missed. Naming them plainly is the credibility of the framework - not hiding them.

What_We_Built_to_Close_Them

I shipped PRs to close the labs twin's three REDs.

Kill switch: A MAINTENANCE_MODE env var checked at all three LLM call sites in engine.ts now gates every new LLM-driven request. Toggled via Railway environment variables - change the flag, next request respects the new state. No code deploy, no rebuild cycle.

Observability: Structured logging with correlation IDs. Every request gets a UUID. Errors are structured JSON with context, not console noise. Per-session Redis counters fire WARN-level structured log alerts on volume spikes, canary leaks, and token thresholds. The audit trail is now persistent and queryable.

Spend: This one remains in progress. The outer wall is confirmed - provider-level caps are configured across the fleet, bounding existential cost risk. The inner wall (in-app visibility and spend tracking) is queued in a follow-up brief. Agent-sdk style cost visibility is the next step: token counts logged per call, daily aggregates, per-user and per-session attribution. Until that lands, the Spend RED stays open.

This is not 'we solved it.' This is 'here is what closed the live REDs, and here is what remains as operational work.'

Seven_Questions_Before_Your_Agent_Ships

The template is downloadable as a standalone checklist at /agent-control-map-checklist.md - seven sections, one per control point, with verdict rows you fill in for your own agent. No login needed. Run it against your own agents.

For each control point, the template asks: what counts as answered? What blocks a green verdict? What is the current state? Answer those rows, assign a verdict, name the gap. Repeat for all seven.

The output is a summary table with a verdict per control point and a fleet-wide verdict. The tool is the table. The table is the audit.

The template does not tell you what order to close gaps in. That is an owner decision - driven by which gaps are live risks and which are operational. But it names the gaps clearly, so the decision is real. You are not choosing between vague concerns. You are choosing between named, enumerable control points.

Auditability_as_the_Floor

An agent shipped without clear answers to these seven questions is an agent you cannot audit when it fails.

An auditable agent is an agent you understand. You know what it costs. You know what it can touch. You know who can invoke it. You know whether you can stop it. You know what trail it leaves when it runs. These are not optional properties. They are the floor.

An agent that hits a bug, accrues unbounded cost, publishes unreviewed content, or runs without a stop mechanism is an agent that has failed structurally - not because of a prompt, but because the operational layer was never closed.

Seven questions before your agent ships.

If you can't answer all seven for any agent you ship, you don't yet have the agent you think you have.