A quarterly AI system audit is not a performance review. It is a scope drift check.
Scope drift is quiet. The system does not break. It just starts handling things it was never designed to handle — via fallback paths, accumulated prompt patches, and retrieval sets that grew without a corresponding accuracy check. By month four, the system is doing something materially different from what it did in month one. Nobody planned that. Nobody documented it.
This audit takes under two hours. Run it every quarter.
What Scope Drift Looks Like in Production
Three patterns show up repeatedly.
Fallback path accumulation. When a query does not match the system's primary routing logic, it falls back to a default handler. That is fine for occasional edge cases. It is a problem when 15% of production queries are hitting the fallback — because that means 15% of queries are outside the original scope and the system is improvising.
Prompt patch accumulation. A prompt fails on a specific input. Someone adds a conditional clause to fix it. Then another. Then another. Six months later, the prompt has 11 conditional clauses, no version history, and nobody can explain what the original logic was. Each patch was reasonable. The aggregate is unmaintainable.
Retrieval set expansion without accuracy checks. The knowledge base grows — new documents, new data sources, new integrations. Retrieval coverage improves. Retrieval precision usually does not. The system retrieves more, but the top-k results for common queries get noisier. Nobody ran an accuracy sample after the expansion.
The Five-Item Quarterly Audit Checklist
1. Scope Boundary Review
Pull the last 30 days of query logs. Categorize each query as in-scope, edge case, or out-of-scope. Calculate the percentage in each bucket. Compare to the same calculation from the prior quarter.
If out-of-scope queries grew by more than 5 percentage points, the system's scope has drifted. Decide whether to expand the defined scope intentionally or add a hard rejection path for the new query types.
Time: 30 minutes.
2. Retrieval Accuracy Sample
Select 20 queries — 10 from the most common query cluster, 10 from the highest-volume edge case cluster. Run each query against the current retrieval system. For each query, manually inspect the top-3 retrieved results and score them: relevant, partially relevant, or irrelevant.
Target: 85% of top-3 results should score relevant or partially relevant. Below that threshold, the retrieval set needs pruning or re-ranking before the next quarter.
Time: 30 minutes.
3. Prompt Version Diff
Compare the current prompt to the prompt from 90 days ago. If you do not have version history, that is the first finding — implement it before next quarter.
For each change, document: what input triggered the change, what the change was, and whether it was tested against a regression set. Any change without a documented trigger or test is a liability.
Time: 20 minutes.
4. Approval Gate Log Analysis
If the system has approval gates — human review steps before outputs are acted on — pull the gate logs for the quarter. Calculate the approval rate, rejection rate, and the most common rejection reasons.
A rising rejection rate signals that the system's outputs are drifting from what operators consider acceptable. A falling rejection rate is not automatically good — it may mean reviewers are rubber-stamping. Both trends need investigation.
Time: 20 minutes.
5. Latency Percentile Comparison Against Baseline
Pull p50, p90, and p99 latency for the current quarter. Compare to the baseline from the system's first month in production.
Latency growth at p99 usually means the system is handling more complex queries, the retrieval set has grown without index optimization, or prompt length has increased from patch accumulation. Each cause has a different fix. Identify which one applies before the next quarter.
Time: 20 minutes.
Running All Five Checks in Under Two Hours
The tooling does not need to be sophisticated.
- Query logs: export from your existing observability stack (Datadog, Grafana, CloudWatch — any of them). Filter by date range. Export to CSV.
- Retrieval accuracy: run queries manually against the production endpoint. Score results in a spreadsheet. 20 queries takes about 25 minutes if you know the domain.
- Prompt diff: store prompts in version control.
git diffbetween the tag from 90 days ago and HEAD is sufficient. - Gate logs: export from whatever system manages your approval workflow. Calculate rates in a spreadsheet.
- Latency: pull from your existing APM tool. p50/p90/p99 are standard metrics in every major observability platform.
The two-hour target assumes the logs are accessible and the person running the audit knows the system. If either condition is not met, fix that first.
What to Do With the Results
The audit produces a short document: five sections, one finding per section, one recommended action per finding. That document goes to whoever owns the system. It is not a report for leadership — it is an operational input for the next sprint.
If the same finding appears in two consecutive quarterly audits without resolution, it is no longer a finding. It is a design flaw. Treat it accordingly.
Systems like AI Brand Presence are built with this kind of operational discipline from the start — defined scope boundaries, retrieval accuracy baselines, and prompt version tracking built into the delivery process, not added later.
Most teams skip the quarterly audit because nothing is visibly broken. That is exactly when scope drift is hardest to see and easiest to fix.