AI EngineeringAI Quality Engineering & Evals
Eval Harness Sprint
A working evaluation harness in the client’s own CI — a reviewed set of test cases, an independent judge and a quality gate that fails the build when quality drops — with baseline scores for every metric they agreed to track, and their own engineers able to run and extend it.
Eval Harness Sprint
$35K–$55K
- Duration: 4 weeks
- Tier: Build
- Format: Fixed scope
You leave withWorking eval harness in the client’s CI with baseline scores for every tracked metric
Book a Free SessionSound familiar?
Every time we change a prompt or upgrade the model, someone tries ten questions and says it looks fine. Last release something that used to work quietly broke and we heard about it from a customer. We have tests for everything else. We want something in the pipeline that fails the build when quality drops — but the answer is different every time, so we don’t know what the test should even look like.
What you leave with
Measurement spec
Format: One page
How we know it’s acceptableAccepted by the product owner in week 1: criteria, exact versus judged checks, thresholds.
Evaluation harness (dataset, judge, runner, adapter)
Format: Code and files in the client’s repository
How we know it’s acceptableRuns end to end in the client’s CI without us; a client engineer can add a case and re-run following the runbook.
Golden dataset
Format: Versioned files in the repository
How we know it’s acceptableEvery case has input, expected answer or rubric, category and difficulty; reviewed by the client’s expert; no placeholder IDs remain.
Baseline report
Format: HTML and JSON
How we know it’s acceptableScores for every metric agreed in week 1, overall, by category and by difficulty; judge agreement with human-marked cases stated.
Runbook
Format: Document
How we know it’s acceptableA client engineer who was not on the project can triage a failed run and add a case by following it; tested in the handover session.
Handover session
Format: Live call, recorded
How we know it’s acceptableAttended by the engineer who will own the harness and by the product owner.
Also included
- A pre-flight check that stops a broken environment being reported as a drop in quality.
- A report for every run, archived so quality can be tracked over time.
How it runs
4 weeks
Week 1
Define “good” and draw the line between exact and judged checks
Criteria for each surface and thresholds agreed with the product owner.
OutA one-page measurement spec; access confirmed.
Week 2
The dataset and the judge
Write or harvest the starting cases with the client’s expert; write the judge and calibrate it against human-marked cases.
OutA versioned dataset and a calibrated judge.
Week 3
Runner, adapter and CI gate
Call the system, run the pre-flight check, report by category, fail the build below threshold, first full run.
OutThe harness running in the client’s CI; the first baseline.
Week 4
Harden and hand over
Tune thresholds against the baseline, check run time and cost, write the runbook, paired handover, and tracing hooks if in scope.
OutBaseline scores for every tracked metric, the runbook, the handover session.
Is this right for you?
This is for you if
- Buyerthe person who owns release quality for the AI feature — a head of engineering, AI or QA, or the product owner accountable for it.
- Also has to say yeswhoever owns the CI pipeline and repository (without them the harness stays a script on someone’s laptop); the product owner, who agrees what counts as acceptable quality and the threshold that fails a build; and Finance, since the band reaches $55K.
This is the wrong answer if
- The system is still changing shape week to week — prompts, tools or models being swapped. We would be measuring a moving target; the Agent Prototype Sprint or a stable release comes first.
- Nobody on the client’s side can say what a correct answer looks like, or no domain expert can give a few hours a week. The dataset is the product. One we write alone encodes our guess, and a gate built on it will be argued with or switched off.
- There is no CI or release process, or we cannot have access to it. A harness nobody runs is only a report; fix the pipeline first, or take the AI Reliability Audit.
- The output is deterministic — a rules engine or a classifier with fixed answers. An ordinary test suite is cheaper and easier to audit.
- They want a one-off view of how reliable the system is today and no ongoing machinery. That is the AI Reliability Audit.
What’s included, and what isn’t
In scope
- one AI system and one entry point
- an offline evaluation harness — dataset, judge, exact checks, runner, reports and a CI gate
- baseline scores
- the runbook
- the handover
Out of scope
- fixing what the harness finds (a build sprint)
- building or changing the AI system
- a second system, priced per surface
- labelling the full dataset (the client’s expert reviews, we write the starting set)
- running the harness, triaging failures and curating the dataset after handover (that is Managed Services)
- adversarial testing (the AI Red Team Sprint)
- the client’s own model-call costs after handover
Before we start
Before day one
- Repository and CI access we can commit to, and a named owner of the pipeline.
- A way to call the system without a user interface (API, batch endpoint or queue) in a test environment configured like production.
- A named domain expert with hours booked for weeks 1–3, and a named technical contact who can answer a question the same day.
- Credentials for a judge model, a budget for run cost, and agreement on whether client data may be sent to it.
- Real examples — logs or traces, or at least real questions and documents — not invented ones.
- A product owner who will set the thresholds by the end of week 1.
What we’ll need from your team
- A domain expert, about 3–4 hours a week in weeks 1–3, to review cases and mark the calibration set.
- A platform or DevOps engineer, about 4 hours a week, rising in week 3 when the CI gate goes in.
- The product owner: one hour in week 1 to set thresholds, a decision within 24 hours when a threshold or scope question arrives, and one hour for sign-off in week 4.
Pricing
What we need to give you a quote
- Which AI systems and surfaces are in scope (chat, agent, extraction) and how each is reached: API, queue or user interface.
- Their CI and repository setup.
- How often prompts or models change.
- Whether real traffic or logs exist to build the dataset from.
- Who the domain expert is and how many hours they have.
- Which judge model they permit, and whether their data may be sent to it.
- The quality bar the product owner has in mind.
- Whether the AI Reliability Audit has already been done.
After we hand over
Who owns it after we leave
A named engineer on the client’s platform or AI team owns the harness, the dataset and the runbook; the product owner owns the thresholds. We say plainly what it displaces: dataset upkeep and triage of failed runs, sized from the week 3 run and stated at handover. Our draft expectation is about half a day a week — to be measured, not assumed.
What we measure
- The baseline accept rate, overall and by category and difficulty
- Judge agreement with human-marked cases
- Time and cost of a full run
Over the following months, measured by the client: regressions caught by the gate before release against regressions found by users.
What comes next
- Managed Services — Growthfor continuous evaluation and dataset curation after handover.
- The AI Red Team Sprintwhose attack suite then runs in the same CI.
- A second harness for another AI surface.
- A build sprint to close the weaknesses the baseline shows.
Questions buyers ask us
You can, and the tools are free. What takes longer is the dataset nobody makes time for, a judge you can trust, the line between what is checked exactly and what is judged, and a threshold your product owner will stand behind. We have built this before and start from that design, so those four are the work, not the plumbing.
Use one; we do. Datasets and traces can sit in it. A platform stores and displays results; it does not know what good looks like for your product. That is the dataset, the judge and the thresholds, and they move with you if you change platform.
Fair. We run the judge at temperature zero, check it against cases a person has marked, report the agreement rate, and use plain exact checks wherever there is one right answer. The runbook includes a regular human spot-check, because a judge is never a perfect oracle.
Every case costs at least two model calls, so the full set does not run on every commit. A small high-signal slice runs on every change and the full set nightly; we size both against your budget and report the cost of a run at handover.
The script is the small part. You are paying for the reviewed dataset, a judge you have checked and a threshold your product owner agreed — the parts that take a team weeks to get right. It is fixed price, and all of it lives in your repository.