What actually breaks when an AI agent runs unattended

The prototype works. It works in the demo, it works for the client on the call, and it quietly falls apart at 2am on a Sunday when a token refresh fails and nobody is watching. That gap — between the demo and week three — is where autonomous agents live or die.

After enough of these deployments we stopped blaming the model. The model is almost never the problem. The problems are boring, structural, and fixable, and they are the same five every time.

1. State that leaks between runs

An agent that remembers something between conversations needs somewhere to put it. The default outcome is a JSON blob in a KV namespace that only one person understands. Within a month you cannot answer "why did it do that", and a prompt change silently rewrites historical behaviour.

What we do instead: give every run a session id, store everything under it, and treat the store as append-only. When a run goes wrong you read the whole trace end to end instead of guessing from a snapshot. It costs a few more reads. It buys every incident investigation we have ever had.

2. Retries that are not idempotent

This is the one that costs money. A tool call charges a card, sends an email, or writes to a contract. The network times out. The agent retries. You now have two charges.

Every non-read tool call gets an idempotency key derived from the run id and the tool call index, and the agent only retries on errors it has been told are safe to retry. Anything that moves value defaults to no retry, surface it. A loud failure at 2am is strictly better than a quiet duplicate at 2am.

tool_call(key="run_8f21::t3::refund")
  -> already completed, returning cached result

tool_call(key="run_8f21::t3::refund")
  -> NETWORK_TIMEOUT
  -> not retryable: value-moving
  -> escalate to human

3. Nobody owns the alert

Plenty of agents have a dashboard. Very few have an alert with a name attached to it. A dashboard nobody watches is a museum.

We insist on three things before an agent goes unattended: a health check that reflects real work done, not just process liveness; an alert that pages a human; and a written runbook that says what to do in the first five minutes. If the runbook does not exist, the agent stays in suggestion mode where a human confirms each action.

4. Guardrails that only exist in the prompt

A rule in the system prompt is a suggestion. Real guardrails are enforced outside the model, in code, on the tool boundary — the place where arguments are validated and permissions are checked.

We validate tool arguments against a schema before the call runs, cap spend per run, and whitelist the endpoints an agent may reach. Prompt text tells the model to be careful. Code makes it impossible to be careless.

5. No ceiling

An agent that loops on a retry, or that decides to research something very thoroughly, or that calls a paid API forty times, will do exactly that. Nobody notices until the invoice arrives.

Hard caps on iterations, tokens, wall-clock time and spend per run, enforced in the orchestrator rather than requested politely in the prompt. When a run hits a cap it stops and reports where it got to. We have watched this single control prevent the majority of runaway bills.