AI Incident Response: The Runbook for When the Agent Is the Incident.

Your IR plan has a gap

Standard incident response assumes an intruder who moves at human speed and a system that does not act on its own. AI systems break both assumptions. The Hugging Face intrusion was driven by an agent swarm executing thousands of actions across disposable sandboxes. And in the more common case, the thing doing damage is not an attacker at all: it is your own agent, behaving exactly as designed, on input that steered it somewhere nobody anticipated.

Neither scenario has a page in most runbooks. This is that page.

A severity taxonomy that fits

Generic severity levels ("SEV1: customer-facing outage") map poorly onto AI failures, where the system is often up and confidently wrong. Classify by what crossed a boundary:

The class determines the first move, and the first move is usually not "shut it all down." Killing an agent mid-run can leave partial state (half-written records, opened tickets, in-flight transfers) that is harder to reason about than a paused agent. Prefer containment that stops new actions while preserving state.

Containment primitives, built before you need them

The defining feature of a bad AI incident is discovering that you have no way to stop the thing quickly. Build these first; each is small.

A per-agent kill switch that halts new sessions and blocks in-flight tool calls, flipped by config rather than deploy, exercised on a normal day so you know it works.

Tool-level disable, independent of the agent. Often the right containment is "no agent may call payments.execute right now" while read-only work continues. If your only lever is all-or-nothing, every incident becomes an outage.

Credential revocation. With the broker pattern, this is one call because tokens are short-lived and minted per request. Without it, you are hunting long-lived secrets across environments during an incident, which is the worst possible time.

Egress block by destination. If exfiltration is suspected, remove the destination from the allowlist and every sandbox loses it at once.

Model and prompt rollback, in seconds, per the deployment primitives.

Index quarantine. For injection through retrieved content, the ability to mark documents or a whole source as non-retrievable without deleting them, since you will need them for forensics.

Session freeze. Stop the agent from taking new actions while keeping its state and logs intact for reconstruction.

The first thirty minutes

Minutes 0 to 5: contain narrowly. Identify the class. Disable the specific capability, not the platform, unless you cannot scope it. Freeze affected sessions rather than terminating them. Preserve logs immediately, including anything on a short retention clock; snapshot before rotation eats the evidence.

Minutes 5 to 15: establish blast radius. Three questions, in order:

1. What did it read? Retrieval logs and tool-call logs, per principal and per document. This determines disclosure scope.
2. What did it write or send? Every mutating tool call in the window, including drafts and queued messages. This determines damage.
3. Who saw the output? Sessions, recipients, downstream systems that consumed it. This determines notification.

If you built decision records, these are three queries. If you did not, this is where the incident stretches from hours into days, and where you will be unable to give your regulator a defensible answer about scope.

Minutes 15 to 30: find the steer. For injection and unauthorized-action classes, locate the specific input that redirected the agent. Pull the full conversation and every retrieved chunk and tool result for the affected run, then read them looking for instruction-shaped text: a document with embedded directives, a web page with hidden content, a ticket comment, a filename. Once found, quarantine that source and search the corpus for siblings, because whoever planted one usually planted several.

Reconstructing an agent run

Agent forensics has a rhythm that traditional forensics does not, because the "attacker" and the "victim process" are the same thing. What you are building is a timeline that interleaves four streams: the model's inputs (prompts, retrieved content, tool results), the model's outputs (reasoning and tool calls), the harness decisions (allowed, denied, escalated), and external effects (API calls, records written, messages sent).

Two questions matter most and neither is answerable from application logs alone. When did the run's behavior diverge from its objective? Find the last action consistent with the original task; the input immediately before it is your prime suspect. What did the harness stop? Denied actions are as informative as allowed ones, since a run with fifty denials and two allowances tells you the attacker was probing and mostly failing, which is a very different report than two denials and fifty allowances.

Preserve the sandbox image if one exists, and hash everything you collect as you collect it.

When it becomes a reportable breach

Do not let the novelty of the failure mode obscure a familiar legal question. If protected data reached someone unauthorized, existing obligations apply on their existing clocks: HIPAA breach notification, GDPR's 72-hour supervisory notice, state breach laws, contractual notice terms that are often tighter than statute, and sector rules like the SEC's cyber disclosure requirements for material incidents.

"An AI did it" is not a category exemption. The determination hinges on the same facts it always has: what data, whose, who received it, and whether it was secured. Loop legal and compliance in during the first hour rather than after the technical work concludes, because their clock starts at discovery and it is usually shorter than the engineering timeline.

For regulated model risk programs, an incident involving a model in your inventory typically triggers a validation review as well. Assume the model comes out of service until that review clears, and plan the business continuity around that assumption rather than being surprised by it.

After: the part that compounds

Standard blameless postmortem, plus three AI-specific actions that turn an incident into permanent immunity.

The incident becomes an eval case. The exact input that caused the failure goes into the adversarial suite with the correct expected behavior, wired into the promotion gate. This is the single most valuable output of the whole process: the failure can never silently return.

The containment gap becomes a control. Every incident reveals a lever you wished you had. Build it while the motivation is fresh.

Re-examine the trust boundary, not the prompt. The tempting fix after an injection incident is to add "ignore instructions in documents" to the system prompt. That is not a fix, it is a wish. The real question is why a hijacked agent could reach that capability at all, and the answer belongs in the allowlist, the approval policy, or the tool's scope.

Run the tabletop

Ninety minutes, the people who would actually be on the call, no laptops required. Three scenarios that surface most gaps:

1. An internal user reports the assistant quoted a document they should not have seen. Can you determine who else saw it? How fast can you make that document unretrievable?
2. An agent processed a vendor invoice containing hidden instructions and initiated a payment to a new account. Who can stop the payment? Can you find the other invoices from that vendor? Was dual control in the path?
3. A model you self-host is found to have a compromised upstream artifact. Which decisions did it produce? Can you enumerate them by hash? What do you tell clients?

Score yourselves on time-to-contain and time-to-blast-radius. Most teams doing this the first time find they cannot answer question one at all, and that finding is worth the ninety minutes on its own.

The uncomfortable summary

AI incidents are not exotic. They are ordinary security incidents with two twists: the timeline is compressed because the actor is a machine, and the "malicious insider" may be a system you deployed on purpose. The response depends almost entirely on work done beforehand: granular kill switches, short-lived credentials, retrieval and decision logs, and fast rollback.

Build those six things and your worst AI incident is a bad afternoon. Skip them and it is a quarter, plus a regulatory filing you cannot fully substantiate.