The first article in this series argued that orchestration is the discipline that closes the gap between AI demos and AI products. This piece is about what happens after the team makes that bet.
The orchestration layer is in place. The graph runs. The first version is in production. And then the second-order problems begin:
- Conversations lose context after a reconnect
- Retrieval returns the wrong chunks for inputs that look almost identical to inputs that worked yesterday
- The QA team has nothing to test against because no one has defined what “correct” means for a subjective output
- Costs surprise the CFO
- A single misbehaving node corrupts state for the entire pipeline
- Recommendations contradict last week’s recommendations for the same input
None of this is unusual. In our experience building production agentic systems across regulated and high-stakes domains, this is the phase where the real engineering begins. The frameworks make orchestration possible. The disciplines covered in this article are what make orchestration reliable.
- What you’ll get from this article
- Why memory has to be partitioned by lifetime, not stored as one undifferentiated blob
- How to build retrieval pipelines that don’t hallucinate at scale
- Why structured outputs are a reliability pattern, not just a feature
- How to embed guardrails inside graph nodes rather than bolt them on
- How to evaluate subjective outputs without spending all your time on manual review
- What to instrument for production observability
- How to engineer the human-in-the-loop pattern at scale
- Five reliability patterns that compound over time
Memory, Partitioned by Lifetime
The single most underrated architectural insight in agent systems is that memory is not one thing. Most teams treat it as a single concern — “we need to give the agent memory” — and end up with a sprawling, unstructured store that grows in unpredictable ways and quietly poisons every downstream decision.
In production, memory has to be partitioned by lifetime. Each partition has its own access pattern, its own retention policy, its own consistency model, and often its own underlying store.
The four partitions and what each is for
Why name spacing is non-negotiable?
The fourth partition — semantic memory — is where most teams get into trouble. The architectural discipline that matters here is namespace isolation. Embeddings should be partitioned per campaign, per analysis, per tenant, or per business context.
- Co-mingling embeddings across tenants creates two failure modes:
- Correctness: one customer’s data leaks into another’s recommendations
- Cost: every retrieval query scans irrelevant vectors, raising latency and bill
Done well, memory partitioning gives a system four properties that compound over time: short-term resumability, durable user-facing history, replayable decisioning, and isolated semantic context. Done badly — or done as a single undifferentiated blob — it gives the system a slow leak that quietly degrades quality until someone is asked to rewrite the memory layer from scratch.
Retrieval Pipelines That Do Not Hallucinate
RAG is not new, but production-grade RAG is harder than the introductory tutorials suggest. The naive pattern — embed every document, retrieve top-k by cosine similarity, stuff the chunks into the prompt — works for small document sets and tightly scoped queries.
It breaks when the corpus grows, when query intent varies, or when the user expects the system to know facts that span multiple sources.
In our production deployments, we treat retrieval as a hybrid pipeline rather than a single step.
The three layers of a production retrieval pipeline
- Proprietary vector retrieval. Handles domain knowledge inside the customer’s own data — historical campaigns, internal policies, prior analyses, product catalogs. Namespaced per tenant. Chunked by semantic unit, not fixed token count. Tuned for recall over precision since we re-rank downstream.
- Web grounding (where applicable). Handles the long tail of facts the model could not be expected to memorize and the customer does not maintain internally. Web search invoked through tooling that supports structured citations. Returned content is treated as untrusted: schema-validated, sanitized, and bounded in size before reaching the next LLM call.
- LLM-driven re-ranking and synthesis. Sits on top of the first two. Retrieved chunks are not stuffed naively into the prompt. They are passed to a dedicated reasoning step that selects the most relevant subset, identifies conflicts, and produces a grounded response with explicit attribution.
What separates production RAG from notebook RAG
- Chunking strategy reflects the structure of the underlying documents, not arbitrary token windows
- Namespace design isolates per tenant or per context
- Re-ranking logic is explicit and tunable, not a hidden default
- Citation enforcement is structural: outputs reference the chunks they used
- Output structure is typed, not free-form
- Teams that skip this discipline end up with systems that hallucinate confidently — which is worse than systems that hallucinate obviously.
- Structured Outputs Are a Reliability Pattern
One of the most consequential architectural decisions in a production agentic system is to treat every LLM-to-system boundary as a typed contract. Free-form text is for users. Everything inside the system is structured.
What this looks like in practice
- Every LLM call that produces output the rest of the system will consume returns a validated schema — through a typed parser that enforces required fields, types, and value ranges
- Validation failures route to a deterministic fallback structure rather than crashing the workflow
- The schema itself is versioned alongside the prompt, so we can trace which schema version produced which output in production
The two failure classes this eliminates
Guardrails Embedded in the Graph
Guardrails in production agentic systems are not a post-processing step. They are domain logic embedded inside graph nodes. The distinction matters more than it sounds.
Post-processing filter vs embedded guardrail
What we typically embed
Geographic constraints — clamp recommendations to the relevant region at the moment of generation Budget bounds — refuse responses that violate negotiated ceilings Brand-safety checks — embedded inside creative-evaluation nodes, not bolted on after Compliance rules — jurisdiction-specific content rules enforced at the moment outputs are produced
In our deployments, this pattern has been particularly important for systems operating in regulated or geographically-bounded domains.
A location intelligence platform serving a single country needs every recommendation clamped to that country at the moment of generation, not filtered after the fact. A healthcare workflow needs compliance rules baked into the consultation summary node itself.
Evaluation: Knowing the System Is Getting Better
Evaluation is the discipline that separates teams who improve their agents methodically from teams who change prompts and hope for the best. It is also, in our experience, the discipline most teams underinvest in until something breaks publicly.
The split: deterministic vs subjective evaluation
Deterministic outputs — classifications, extractions, structured fields — are tractable. Build a labeled test set, run the pipeline, measure precision and recall, track regressions in CI. The infrastructure here is well-understood.
Subjective outputs — strategic recommendations, generated content, qualitative scores, conversational responses — are harder, and most off-the-shelf approaches fall short. The three techniques we use for subjective evaluation
1. Structured rubrics. Subjective tasks are decomposed into scorable attributes. For a creative output, this might mean brand alignment, audience fit, platform suitability, and authenticity, each scored on a defined scale with explicit anchors.
This converts a single fuzzy judgment (“is this creative good?”) into a multi-attribute score that can be averaged, tracked over time, and disagreed about coherently.
2. LLM-as-judge frameworks. A separate LLM, with a different prompt and often a different model, scores the output of the production system.
We treat these scores as one signal among many. They correlate well with human judgment for many tasks, but they have known biases (length bias, position bias, self-preference) that have to be controlled for. Calibration against a human-labeled subset is non-negotiable.
3. Regression suites running against golden datasets. Curated inputs with expected output ranges, run on every meaningful change.
The first time a prompt change improves average score but silently breaks behavior on a specific edge case, the value of the regression suite becomes obvious.
None of this is glamorous. All of it compounds. Teams that take evaluation seriously ship better agents over time. Teams that do not, do not.
Observability: Traces, Versions, and Debugging in Production
Production agentic systems are stochastic.
Even with structured outputs, embedded guardrails, and rigorous evaluation, the same input can produce different outputs across runs. When something goes wrong, the engineering team needs to be able to reconstruct exactly what happened.
What every LLM call should capture
- The prompt version
- The input state
- The model and parameters used
- The retrieval set fed in
- The structured output produced
- The latency and token cost of the call
Traces are linked across nodes in a graph invocation, so a single workflow execution can be replayed end-to-end.
What this enables in production
Debugging. When a user reports a problem, the engineer pulls the trace and sees exactly what the system did.
Regression analysis. When a prompt change ships, traces from before and after the change are compared to identify behavioral drift.
Cost and performance monitoring. Token spend and latency profiles per node make it possible to identify which parts of the graph are worth optimizing.
The discipline that holds it together
Prompt versioning. Prompts are not strings inlined into agent code.
They are versioned artifacts, registered in a prompt store, with the active version recorded in every trace. When a regression appears, we know which prompt version produced it. When an experiment ships, we know which traces correspond to which arm.
The Human-in-the-Loop Pattern
For any agentic system operating in a domain with consequential decisions, human checkpoints are not optional. They are an architectural feature.
The wrong way to do human review
- A separate workflow alongside the automated pipeline
- A Slack notification, a side dashboard, an email queue
- The reviewer sees only the candidate output, not the upstream context
- The approval is captured outside the system in an unstructured way
- Reviewer decisions never feed back into evaluation or training
- This works at small scale and breaks at any meaningful volume.
The pattern that scales
- Make human review a node in the graph. The node:
- Pauses execution at a defined decision boundary
- Surfaces the relevant context — input, candidate output, upstream traces, agent’s reasoning
- Waits for an approval, modification, or rejection
- Resumes the graph with the human’s decision recorded as part of the durable state
- This pattern uses LangGraph’s checkpointing primitives directly. The graph is paused, the checkpoint persists, and the resume operation picks up exactly where the pause occurred.
Why this compounds
- Reviewer decisions become labeled data for evaluation
- Agents that frequently get overridden in a particular pattern surface as candidates for prompt revision
- The audit trail captures not just what the system did, but who approved it and why
- A reviewer can take their time, come back the next morning, and resume exactly where they left off
Reliability Patterns That Compound
Across our production deployments, a small set of reliability patterns recur. They are not flashy. They are the patterns that quietly turn a fragile system into a sturdy one over the course of a year of operation.
The five patterns
1. Idempotent recompute via hash lookup. Persist outputs keyed by a hash of the input plus configuration. When the same input arrives again, short-circuit the full graph run and return the prior result.
Repeat-workload latency drops from tens of seconds into single-digit seconds. Token spend drops accordingly.
2. Early-return with background enrichment. Long-running enrichment should not block the user-facing response. The graph returns an initial result as soon as the fast path completes, and continues enrichment in the background.
The user sees the system as responsive. The durable record gets the deeper analysis a few seconds later.
3. Deterministic fallbacks for malformed responses. Every LLM call has a fallback path. If the response fails schema validation, the node returns a deterministic structure that allows the pipeline to continue with degraded but valid output.
The fallback is logged, traced, and surfaces as a quality signal in evaluation, but it does not crash the workflow.
4. Partial-pipeline persistence on non-fatal errors. When a non-critical node fails (an enrichment, a secondary analysis, an optional step), the pipeline persists what it has and continues. The user gets the core output. The enrichment is retried asynchronously or surfaced as missing.
5. Fail-fast startup checks. On boot, the service validates that every external dependency is reachable and configured — database connections, vector store, model API keys, observability backend.
A missing API key surfaces in seconds at startup rather than as a cryptic 500 ten minutes into a customer’s first session.
Each pattern is small. Together, they are the difference between a system that operates as steady infrastructure and a system that creates a constant stream of incident reports.
The Three Things to Remember
If you take only three things from this article, take these:
- Memory must be partitioned by lifetime. Short-term working state, durable transcripts, append-only long-term context, and namespaced semantic memory each have their own access patterns and stores. Treating memory as one undifferentiated thing creates a slow leak that degrades quality over time.
- Every LLM-to-system boundary is a typed contract. Structured outputs eliminate an entire class of production failures. Free-form text is for users. Everything inside the system is schema-validated.
- Guardrails belong inside the graph, not after it. Embedded constraints catch violations at the source, cost nothing extra, and cannot be silently bypassed. Post-processing filters are a failure mode disguised as defense in depth.
The teams at BigStep have spent the past several years operating production agentic systems through every one of the failure modes described above. The patterns in this article are the ones we have learned to lean on.
Frequently Asked Questions
How do we decide which memory partition a particular piece of state belongs to?
The decision is driven by lifetime and access pattern. State that is needed only for the duration of a single workflow execution belongs in the checkpointed working state. State that the user will see again belongs in the durable transcript.
State that future workflow runs may want to reuse belongs in the append-only long-term context. State that needs to be retrieved by semantic similarity belongs in the vector store. When in doubt, partition more rather than less.
Merging partitions later is straightforward; splitting them after the fact is painful.
Should we use a single vector database or multiple specialized stores? For most production systems, a single well-namespaced vector database is sufficient and operationally simpler.
The discipline that matters is not the number of stores; it is per-tenant or per-context namespacing within whatever store is chosen.
Multiple stores become justified when access patterns diverge sharply — for example, low-latency real-time retrieval versus large-scale batch analytical retrieval.
How do we evaluate subjective outputs without spending all our time on manual review?
The cost of evaluation drops sharply once structured rubrics are in place. We typically build a small human-labeled gold set (a few hundred examples), use it to calibrate an LLM-as-judge harness, and then run the harness automatically as part of the regression suite.
Manual review becomes targeted at edge cases the harness flags as uncertain, rather than uniform sampling.
What is the right balance between retrieval and the model’s parametric knowledge?
For any fact that changes over time, that lives inside the customer’s proprietary data, or that the model cannot be reliably expected to memorize, retrieval should be the source of truth.
For general reasoning capabilities — language understanding, summarization, structured generation — the model’s parametric knowledge is what you are paying for.
The mistake we see most often is over-retrieving: stuffing the prompt with chunks the model did not need, which raises cost and often degrades output quality.
Where should guardrails live if the same constraint applies to multiple agents?
Common guardrails belong in a shared library invoked from every relevant node, not duplicated across nodes. The constraint is enforced inside each node’s logic, but the implementation is centralized.
This pattern keeps the guardrails consistent across agents and makes policy updates a single change rather than a cross-cutting refactor.
How do we handle the cost surprise that often hits the first time a multi-agent system goes live? Cost discipline is largely a matter of three patterns: idempotent recompute via hash lookup (do not pay twice for the same input), aggressive structured-output discipline (do not generate output tokens you cannot use), and per-node model routing (cheap models for classification and routing, premium models for reasoning).
Together, these usually bring costs into a predictable range. The third article in this series goes deeper on cost engineering at enterprise scale.
What is the most important observability signal to instrument first? If a team can only instrument one thing, it should be the prompt-versioned trace: every LLM call captures the prompt version, the input, the model, the structured output, and the latency.
This single signal supports debugging, regression analysis, and cost monitoring, and it is the foundation that every other observability investment builds on.
Building Production-Grade Agentic Systems?
If your team is operating a multi-agent system in production and running into the second-order problems described in this article — memory drift, retrieval inconsistency, evaluation gaps, observability blind spots — we would be glad to talk.
The BigStep AI engineering team helps organizations harden their agentic systems against the failure modes that surface only at scale. Get in touch with our AI engineering team →
