AI EngineeringAI Quality Engineering & Evals
Agent Reliability Scorecard
A score across five areas — reliability, evaluation coverage, security, cost and observability — for the AI system they already run, with the biggest gaps named and ranked and a recommended first step, from 15 questions and about an hour of their time.
Agent Reliability Scorecard
Free
- Duration: 60 minutes
- Tier: Start
- Format: Working session
You leave withA score across five areas (reliability, evaluation coverage, security, cost and observability), with the biggest gaps named and ranked and a recommended first step
Book a Free SessionSound familiar?
We launched the assistant a few months ago and it seems fine, but I could not tell you how good it actually is. Every month someone asks whether it is production-ready and I don’t know what I would measure. We have a few dashboards and a QA team that tries it before releases. I have no idea where the gaps are, or which one to fix first.
What you leave with
Scorecard report
Format: PDF
How we know it’s acceptableScores for all five areas and overall; every gap has a one-line reason and a rank. An unanswered question shows as “no answer” and scores low, never blank.
Recommended next step
Format: One section of the report
How we know it’s acceptableNames one programme, or “none”, tied to a specific gap; in the assisted version, reviewed by the engineer who ran the session.
Written summary (assisted only)
Format: One page, PDF
How we know it’s acceptableSent within two working days; readable by a non-technical executive.
Also included
- A comparison with what we would expect of a system of this kind.
- Assisted version only: a 60-minute walkthrough with a BigStep engineer and a one-page written summary within two working days.
How it runs
60 minutes
Assisted, before
About 30 minutes preparing from whatever the client has shared.
OutA short list of questions to probe.
Assisted, session (60 minutes)
Walk the 15 questions with the client’s engineer, ask for evidence behind any high score, and score in the room.
OutScores agreed on the day.
Assisted, after
Within two working days.
OutThe report and a one-page summary naming the ranked gaps and a recommended next step.
Is this right for you?
This is for you if
- Buyerthe person accountable for a live AI system behaving well — a head of AI or engineering, a CTO, or the product owner who takes the complaints.
- Also has to say yesfor the assisted session, the engineer who actually knows how the system is traced, tested and costed has to be in the room, and the client’s security lead if the security questions are to be answered properly.
This is the wrong answer if
- The system is not live yet. The questions are about a running deployment; for an idea or a prototype the Possibilities Session or the Agent Prototype Sprint is the right start.
- The prospect wants something measured or tested. The Scorecard scores what they tell us and touches nothing. Evidence is the AI Reliability Audit; an attack on the system is the AI Red Team Sprint.
- The prospect wants a certificate, an audit opinion or a ranking against named competitors. It is a self-assessment and must never be read as any of those.
What’s included, and what isn’t
In scope
- one live AI or agent deployment
- 15 questions across five areas
- a score per area and overall
- gaps ranked
- a recommended next step
- Assisted version: one 60-minute session and a one-page written summary.
Out of scope
- access to the client’s system, code, data or traces — we look at none of it
- testing or verification of any answer
- a certificate, an audit opinion or a named-competitor benchmark
- a second system (that is a second Scorecard)
- fix advice beyond the one-line next-step pointer
- a repeat session (re-scoring after fixes is what the Audit is for)
Before we start
Before day one
- Assisted: a live AI or agent deployment — not a plan or a prototype.
- The client’s engineer who knows how it is built and monitored is booked into the session, and has had the five-area pre-read so they can find answers beforehand.
- Agreement that we see no system, data or code; we score what we are told.
What we’ll need from your team
- One engineer who knows how the system is built and monitored: the 60-minute session, plus about 30 minutes beforehand to find answers.
- Optional: the security lead for 15–20 minutes if the engineer cannot answer the security questions.
Pricing
Free
After we hand over
What we measure
- Scorecards completed and assisted sessions held
- The share that end with a recommended next step
- The share that become a paid Audit, Harness or Red Team conversation within a set period
The average score per area across all completions is also what turns “benchmark” into something real.
What comes next
- The AI Reliability Auditfor most — it measures what the Scorecard only asked about.
- The Eval Harness Sprintwhere the main gap is that nothing is measured in CI.
- The AI Red Team Sprintwhere security scores lowest and the system has tools or sensitive data.
- The Agent Prototype Sprintif the “live deployment” turns out to be nearer a prototype.
Questions buyers ask us
It is how we meet people. The report may well recommend one of our paid programmes, and it will also say so when the right answer is to do nothing. The report is yours either way.
Fair, and the report says so: it shows where you are weak, not how weak. That is the reason the Audit exists. Every score above the midpoint asks for one piece of evidence, so a high score on something you have never measured shows up as a gap.
Good; the questions ask what they cover. A normal test suite shows the system runs. The Scorecard asks whether anyone could tell that answers got worse after a prompt or model change. Tracing shows what happened, not whether it was any good.
They are not meant to. Fifteen questions across five areas find where to look first; the output is a ranked list, not a verdict.
Proposed: we use them to produce your report and follow up once; we do not publish them; anonymised scores feed our benchmarks only if you opt in.