Audit Trails for AI Decisions: The Evidence Architecture.

The question that ends the meeting

An examiner, an auditor, or a client's risk team points at an output from four months ago and asks: what produced this, what did it see, who approved it, and how do you know?

For traditional software, you answer by reading the code and re-running it. For AI systems that answer does not exist. Temperature is nonzero, the model version may have changed, the retrieval corpus has moved on, and the vendor may have deprecated the endpoint entirely. You cannot reproduce the decision, so the log has to be the record. That single constraint drives the whole architecture.

Most teams discover this backwards. They instrument for debugging, capturing prompts and latencies into an observability tool with 30-day retention, and then find that the thing an auditor wants is a durable, tamper-evident, queryable record of decisions, joined to the business artifacts they affected.

What a decision record contains

The unit is not "an LLM call." It is a decision: one thing the system concluded or did that a person or process relied on. A decision may involve several model calls, several retrievals, and a human approval.

@dataclass(frozen=True)
class DecisionRecord:
    # Identity and lineage
    decision_id: str            # ULID; referenced by the business artifact
    trace_id: str               # joins to your APM traces
    parent_id: str | None       # for multi-step agent runs

    # Who
    principal_id: str           # authenticated human or service, never model-supplied
    on_behalf_of: str | None    # if acting for a client or another user
    tenant_id: str

    # What the system was
    model_id: str               # "internal-general-v3"
    model_hash: str             # sha256 of the served artifact. The alias can move.
    prompt_template_id: str     # "credit-memo/v7"
    prompt_template_hash: str
    params: dict                # temperature, top_p, max_tokens, seed if set
    tool_schema_hash: str       # tool definitions available at call time

    # What it saw
    input_digest: str           # sha256 of rendered input; content stored separately
    retrieved: list[dict]       # [{doc_id, chunk_id, version, score, acl_decision}]
    tool_calls: list[dict]      # [{name, args_digest, result_digest, latency_ms}]

    # What it produced
    output_digest: str
    output_ref: str             # pointer into the content store
    finish_reason: str
    tokens: dict

    # What happened next
    human_action: str | None    # approved | rejected | edited | none
    human_principal: str | None
    human_at: str | None
    business_ref: str | None    # invoice id, memo id, ticket id

    # Integrity
    recorded_at: str            # server time, not client
    prev_hash: str              # chain link
    record_hash: str

A few of these fields are the ones teams regret omitting.

model_hash, not just the alias. If applications call internal-general-v3 and you repointed that alias in April, then every record that stores only the alias is ambiguous about what actually ran. The hash of the served artifact removes the ambiguity permanently.

prompt_template_hash. Prompts are code and they change more often than models. "Which version of the prompt produced this" is a question you will be asked.

retrieved with ACL decisions. This is what proves the answer was grounded only in documents the user was entitled to see, and it is the difference between a plausible defense and an evidenced one. It comes straight out of permission-aware retrieval.

human_action and business_ref. The join between the AI system and the world. Without it you have a pile of model calls; with it you can answer "show me every decision that fed a signed credit memo in Q2."

Digests in the record, content beside it

Prompts and completions in regulated settings contain client data. Putting them inline in an append-only, long-retention store creates a permanent copy of sensitive content in a system designed to resist deletion, which collides directly with deletion obligations.

The pattern that resolves it: the decision record stores hashes and a pointer; the content lives in an encrypted content store with its own lifecycle, its own key, and its own access controls.

The hash is what gives this integrity. If the content is later produced in a dispute, anyone can verify it matches what was recorded at the time. If content must be deleted under a client's data-return clause or a privacy request, the content goes and the record survives with proof that something specific existed, which is usually exactly what regulators and courts want.

Tamper evidence

An audit log that the application can rewrite is not evidence. Two mechanisms, used together:

Hash chaining makes any retroactive edit detectable, because every record commits to its predecessor:

def seal(record: dict, prev_hash: str) -> dict:
    body = {k: v for k, v in record.items() if k != "record_hash"}
    body["prev_hash"] = prev_hash
    canonical = json.dumps(body, sort_keys=True, separators=(",", ":"))
    body["record_hash"] = hashlib.sha256(canonical.encode()).hexdigest()
    return body

def verify(chain: list[dict]) -> int | None:
    """Returns the index of the first broken link, or None if intact."""
    prev = GENESIS
    for i, rec in enumerate(chain):
        if rec["prev_hash"] != prev:
            return i
        expected = seal({k: v for k, v in rec.items()
                         if k not in ("record_hash", "prev_hash")}, prev)
        if expected["record_hash"] != rec["record_hash"]:
            return i
        prev = rec["record_hash"]
    return None

Publish the head hash somewhere outside the system on a schedule (a separate account, a signed daily digest to your GRC platform) so that even wholesale chain replacement is detectable.

Write-once storage makes edits impossible rather than merely visible. S3 Object Lock in compliance mode, in a separate logging account that operational roles cannot write to. Broker-dealers subject to SEC Rule 17a-4 will recognize the shape of this requirement; the 2022 amendments allow either a write-once-read-many approach or an audit-trail alternative, and the chained-plus-locked design satisfies the spirit of both. Confirm the specific retention period and format with your compliance function rather than inferring it from a blog post.

Retention, decided once and enforced

Three clocks usually apply and they rarely match: your regulatory retention (often years), your privacy obligations (delete on request, minimize by default), and your engineering instinct (30 days, because storage costs money).

Resolve it explicitly by tier. Decision records are small, structured, and cheap; keep them for the full regulatory period. Content is large and sensitive; keep it for the shorter of the contractual and regulatory window, with deletion that actually fans out. Debug telemetry is neither evidence nor obligation; expire it fast.

Writing this down as a table with an owner per row is a half-day of work that answers a question every audit asks.

Querying it under pressure

The test of an evidence architecture is how fast you can answer these:

• Every decision produced by model hash abc123 between two dates, and their outcomes.
• Every decision that retrieved document doc-9f2, and which principals saw it.
• Every decision a given principal approved in Q2, and the diff between model output and final artifact.
• Every decision where the human rejected or edited the output, which is your best unsupervised signal of model quality drift.
• The full chain for one decision: inputs, retrievals, tool calls, output, approval.

If those take an engineer a week of log spelunking, you do not have an audit trail, you have logs. A modest columnar store or a partitioned Parquet layout in S3 queried with Athena handles this at a cost that rounds to nothing next to the alternative.

That fourth query is worth building a dashboard for even if nobody ever audits you. A rising human-edit rate is a quality regression that no offline eval caught, and it is the earliest warning you get.

The pleasant side effect

Teams build this because a regulator or a client demands it, and then find it pays for itself in incidents. When something goes wrong, the first thirty minutes are entirely about reconstructing what the system saw and did. A decision-record store turns that from archaeology into three queries.

It also produces your compliance evidence as a byproduct of running the system, rather than as a quarterly scramble to assemble screenshots. That is the same argument as the eval gate: build the control into the pipeline, and the artifact the auditor wants falls out of normal operation.

Start smaller than the schema above

You do not need all of it in week one. The minimum that is genuinely useful: decision id, principal, model hash, prompt hash, input and output digests with content stored separately, retrieved document ids, human action, business reference, and hash chaining into locked storage.

That is a day of work for one engineer, and it is the difference between answering the examiner's question in an afternoon and explaining why you cannot.