AI EngineeringAI Quality Engineering & Evals
AI Reliability Audit
A scored reliability report on their live AI system — how often it is right, where it fails, whether anyone could tell if it got worse, and what it costs per task — measured on a sample of their real outputs, plus a remediation plan ordered by what to fix first.
AI Reliability Audit
$18K–$30K
- Duration: 2 weeks
- Tier: Diagnose
- Format: Fixed scope
You leave withScored reliability report on a live AI system with a prioritised remediation plan
Book a Free SessionSound familiar?
Users tell us the answers are sometimes wrong, but nobody can say how often. We changed the prompt last month and then the model, and we have no way of knowing whether it got better or worse. Before a release, somebody tries a few questions and says it looks fine. Leadership wants a number and I don’t have one.
What you leave with
Measurement spec
Format: One page
How we know it’s acceptableAccepted by the client in week 1: tasks in scope, what “good” means, exact versus judged checks, thresholds.
Scored reliability report
Format: PDF and Word
How we know it’s acceptableEvery score traces to cases in the sample; every failure example shows input, output, expected answer and cause; judge agreement with human-marked cases is stated.
Starter test set
Format: Versioned file (JSON or spreadsheet)
How we know it’s acceptableEach case has input, expected answer or rubric, category and difficulty; reviewed by the client’s expert; re-runnable.
Remediation plan
Format: Spreadsheet
How we know it’s acceptableEvery item has a severity, an effort estimate and a suggested owner; walked through at the readout.
Executive summary
Format: One page
How we know it’s acceptableReadable by a non-technical executive; approved by the buyer.
Readout session
Format: Live call, recorded
How we know it’s acceptableAttended by the engineering owner and the buyer.
Also included
- A measured baseline: the share of answers a user would accept, overall and by category and difficulty, with real failure examples.
- Failure analysis: each failure traced to its cause — the model, the system around it (retrieval, tools, data) or the environment.
- Coverage and cost review: what is tested today against what real traffic looks like, what the system costs per completed task, and what the team could not see because it was not traced.
How it runs
2 weeks
Week 1, days 1–2
Scope and define “good”
The tasks in scope, what a good answer is for each, which outputs are checked exactly and which are judged, and the thresholds the client would accept.
OutA one-page measurement spec, agreed with the client.
Week 1, days 3–5
Assemble the sample from real traffic, agree expected answers or rubrics with the client’s expert, and check access works.
OutThe reviewed sample, as a versioned file.
Week 2, days 1–3
Score the sample with the judge and exact checks, calibrate the judge on a human-marked subset, and review tracing, coverage and cost.
OutScored results and a failure log.
Week 2, days 4–5
Triage each failure as model, system or environment; write the report, remediation plan and executive summary; readout.
OutThe report, plan, summary and a recorded readout.
Is this right for you?
This is for you if
- Buyerthe person accountable for a live AI system behaving well — a head of AI or engineering, a CTO, or the product owner who takes the complaints.
- Also has to say yesthe system owner, who gives access to traces, logs and sample outputs; whoever owns the data, because real outputs may contain customer data (security or compliance); and Finance.
This is the wrong answer if
- The system is not live. The Audit scores real outputs. A demo, or a prototype tested on invented inputs, gives a number nobody should trust; the Agent Prototype Sprint comes first.
- The client already knows it is unreliable and wants it fixed. Do not sell a diagnosis for a problem they have diagnosed; the Eval Harness Sprint or a build sprint is the right start.
- We cannot get any real outputs or traces, even a small redacted sample. We do not score on invented inputs.
- The worry is attack, not accuracy — data leaks, prompt injection, misuse of tools. That is the AI Red Team Sprint.
- The client wants a certificate or an audit opinion. We are not an auditor, and a scored report will be read as one.
What’s included, and what isn’t
In scope
- one AI system
- a sample of its real outputs
- a scored report across reliability, evaluation coverage, cost and observability
- failure analysis
- the remediation plan
- the executive summary and readout
- Number of distinct tasks included at the base price: upto 3.
Out of scope
- building the evaluation harness or anything in the client’s CI (that is the Eval Harness Sprint)
- fixing the failures (a build sprint)
- adversarial or security testing
- certificates or audit opinions
- load or performance testing
- monitoring after the readout
- a second system
- the client’s experts’ time, which is theirs to give
Before we start
Before day one
- A sample of real requests and answers, logged or traced and redacted if needed, with approval from whoever owns the data.
- The prompts, tool definitions, model and version, and any existing tests and dashboards.
- A named domain expert, with hours reserved, who can say whether an answer is right.
- A named technical contact who can answer a question the same day.
- Someone with authority to decide what counts as acceptable, available in the first two days.
What we’ll need from your team
- A domain expert, about 3–4 hours a week, to agree expected answers and mark a calibration set.
- A platform or AI engineer, about 3–4 hours a week, for access, traces and architecture questions.
- A decision-maker for one hour in week 1 to agree what counts as acceptable, available within 24 hours when a scope question arrives, and at the readout.
Pricing
What we need to give you a quote
- What the system does and which models it uses.
- Roughly how many requests a week.
- Whether requests and answers are logged or traced, and in which tool.
- Whether it uses tools or retrieval.
- How many distinct tasks or surfaces would be scored.
- Who can say whether an answer is right, and how many hours a week they could give.
- How sensitive the outputs are (personal, financial, health data) and who must approve sharing a sample.
- What prompted this — a complaint, a model change, a board question.
After we hand over
What we measure
- The share of answers accepted on the sample, overall and by category and difficulty, as the baseline
- Cost per completed task
- The number of evaluation-coverage gaps found
- Judge agreement with human marks
Ask the client at the start to write down their own current answer to “how often is it right?” — for most the answer is no number — so the Audit is the first one.
What comes next
- The Eval Harness Sprintto turn the baseline into a gate in the client’s CI — a cleaner job because the measurement spec and starter test set already exist.
- A build sprintfor the highest-severity fixes where the client has no capacity.
- The AI Red Team Sprintwhere the Audit shows tools or sensitive data that nobody has attacked.
- Managed Services — Growthfor continuous evaluation after that.
Questions buyers ask us
Good, and they become part of the sample. A fixed scenario checks one answer; AI output varies. We ask a different question: across a representative sample of real traffic, what share of answers would a user accept, and has that moved?
If you know what to measure and have the people, do that: go to the Eval Harness Sprint. The Audit is for the team that cannot yet say what “good” is, how often it fails or which failures matter. In two weeks it tells you what the harness should measure, so you do not build the wrong one.
Fair. A judge is a model and can be wrong. We run it at temperature zero, check it against answers a person has already marked, report how often they agree, and use plain exact checks wherever there is one right answer, such as a number or a date. If agreement is low, the report says so.
Yes: it measures today and gives you a baseline for tomorrow. To keep measuring on every change, take the Eval Harness Sprint, or have it run continuously under Managed Services — Growth.
We do not need all of it. We work from a sample of requests and answers that you choose and redact, under your data agreement. If not even a redacted sample is possible, the Audit is the wrong programme.