AI Readiness Audit: What to Check Before Production

An AI readiness audit is a structured decision process, not a certificate and not a generic maturity score. Its purpose is to make the next step explicit: build, narrow the scope, harden an existing system, or stop until a missing dependency is resolved.

Start with the workflow and the decision

Describe the job the system should perform, who is affected, what the output changes, and what happens when the system is uncertain. A readiness review that starts with a model catalogue usually misses the operational boundary. The first artefact should be a workflow map with inputs, transformations, approvals, side effects, and rollback points.

Define a baseline before testing. It may be a manual process, an existing rule, or a small verified dataset. The baseline need not be perfect; it needs to be observable and relevant to the decision.

Data access and quality

Inventory source systems, owners, formats, update frequency, retention, and access paths. Check whether the system receives the data it is authorised to use, not merely whether an API can technically return it. Record missing fields, conflicting definitions, stale records, units, language variation, and duplicate identifiers.

For retrieval systems, inspect chunking, metadata, permission filters, provenance, and behaviour when no relevant source is found. For generated reports, inspect number grounding and reconciliation. A useful acceptance question is: can a reviewer trace a consequential output to the source records that supported it?

Identity, security, and legal boundaries

Map users, service identities, tools, providers, logs, and third-party egress. Check least privilege, secret handling, prompt-injection exposure, output validation, and redaction before data reaches a provider or a log. Test high-risk tool calls with explicit abuse cases; a policy document alone is not evidence that a boundary holds.

Legal and compliance questions should be routed to the responsible specialists. The technical review can document purposes, data categories, retention, human oversight, and vendor terms, but it is not individual legal advice. Record unresolved questions as delivery dependencies rather than hiding them in a risk register.

Model, provider, and evaluation choices

Choose a model against the workflow’s quality, latency, cost, data, and operational constraints. Provider choice is a changeable architecture decision, not the product definition. Pin versions where reproducibility matters and document fallback behaviour.

Build a representative evaluation set with critical, ordinary, ambiguous, and adversarial cases. Define scoring criteria before looking at results. Combine deterministic checks—schema, permissions, figures, citations—with human review for qualities that require judgement. A release gate should state what blocks deployment and who can approve an exception.

Observability and operations

Specify request identifiers, prompt and model versions, token and cost dimensions, latency, validation results, tool calls, and human outcomes. Decide what is retained and what is redacted. Create an incident path for provider errors, unsafe output, data-boundary violations, quality regressions, and cost anomalies.

The minimum operational package is a dashboard that answers known questions, an alert catalogue, a runbook, a rollback path, and an owner. If no one can operate the system after handover, readiness has not been demonstrated.

Go, narrow, or no-go

A go decision requires a bounded workflow, usable data, an accountable owner, tested controls, a relevant evaluation baseline, and an operating path. Narrow the scope when quality is acceptable only for a subset of cases or when a human gate is essential. A no-go is appropriate when identity, data rights, source quality, or incident ownership is unresolved.

The output of the review is a prioritised action plan with evidence, assumptions, open questions, and acceptance criteria. That makes the next engineering step smaller and safer. The Engineering service and Security service describe how those actions can be implemented.

Make the evidence usable

Every finding should state the observation, the consequence, the evidence behind it, the owner, and the condition for closure. “Improve security” is not an action; “tool X has write access without a confirmation state; remove the permission or add a reviewed gate” is. Keep assumptions visible, especially when a source system, provider contract, or retention rule has not yet been verified.

The audit can be run as a short sequence: align on the decision, map the workflow and data, test the critical boundaries, evaluate representative cases, then agree the delivery order. The result is useful even when the answer is no-go because it shows exactly what must change before another investment is sensible.