When Your LangGraph Pilot's Lift Is Just Noise

The read-out was flat. The room split into the two explanations always available after a null: the model wasn't good enough, or the test was too small. Both assume the experiment was about the model. Both usually miss the point. The forecast, written down before the run, said every candidate was predicted to move the metric by less than the design could resolve.

LangGraph in operations, a 10-part series. ← Part 4 · Part 6 →

The Null Was Already in the Checkpoint Before the Node Ran

The pattern is a step in the StateGraph that runs ahead of the acting step and records the movement expected from each candidate and the smallest change the design can see. The checkpointer persists both on the run's thread, and the review can read the forecast out of the checkpoint history rather than out of someone's recollection of the kickoff call.

The persistence documentation describes the mechanism: a checkpointer writes a thread's graph state as checkpoints, so what a run wrote survives the run. Time travel closes the loop, because get_state_history returns every checkpoint on the thread, letting the state before a step be set beside the state after it.

The Constraint Was Named Before the Test, and It Wasn't the Thing Being Optimised

The constraint was written down before anyone touched a prompt: work available multiplied by the share of that work someone takes up. The programme that ran was a contact programme, a take-up lever, which can only move a product whose other factor is non-zero and variable.

The diagnostic to run before the pilot: write the constraint as a product, then ask which factor your lever moves.

A Test That Cannot See the Effect Reports Zero, Not "No Effect"

With a small sample and a noisy metric, the smallest effect a test can detect is often several times larger than any effect the intervention could plausibly produce. The affordable way to lower that floor is to reduce variance rather than raise n — the direction Poyarkov et al. (2016) take in their boosted-decision-tree regression adjustment for online controlled experiments.

In this pilot every candidate was predicted to move the metric by less than the test could see. The null was the forecast; a hit would have been the surprise.

The Lever Everyone Wanted to Pull Had No Path to the Number

The modelled uplift from contact was indistinguishable from zero, for a mechanical reason: the contractors being contacted claim work whether or not anyone contacts them. The behaviour the programme intended to nudge was already at ceiling, so the effect of "message delivered" on "work taken up" was indistinguishable from zero.

A Channel That Reaches Almost Nobody Is Not a Wording Problem

The voice channel reached a person in a small single-digit percentage of calls. The text channel produced no replies at all. No amount of LLM polish rescues a delivery path that terminates before a human is involved.

Most of the Loss Was Where Nobody Went

The large majority of lost orders sat in areas nobody covered, not in the refusals the team was tracking. That is a coverage problem, and no wording change reaches it.

Write the Forecast and the Resolution Floor Into State

The fix is small and architectural. A forecast node writes the predicted movement for each candidate, and the smallest change the design can see, into state before the acting node runs; the checkpointer keeps both on the run's thread, so the review reads them back next to the outcome.

Note what is not stored. The outcome is derivable from telemetry, so under LangGraph's own rule it gets computed when needed. Thinking in LangGraph is explicit that data you cannot reconstruct later must be kept. Keep the forecast; compute the outcome. Then wire the read-back into LangSmith, so the comparison becomes a query rather than a meeting.

Your LangGraph Agent Manufactures the Variance You Then Read as Signal

Every graph-based agent has a noise floor, and most pilots measure against a baseline with a different one.

The New Computer team, in LangSmith's experiment loop, reports 50% higher recall and 40% higher precision against their previous dynamic-memory baseline, with regressed runs inspected individually.

Outcome-Only Evaluation Cannot Tell You Which Lever Moved

Almost every LangGraph pilot is evaluated by the final number. Liu et al. (2025) argue against that: encoding trajectories as graphs and applying process-level metrics shows that richer prompts and stronger models produce structurally different exploration, not merely different scores.

And read your own variance against a ceiling: Ma et al. (2026) built the Data Agent Benchmark, where the best state-of-the-art agent reaches only 57% pass@1. When the best system leaves 43 points of headroom, most of your spread is capability. When your delivery path barely reaches anyone, none of it is.

What to Ask Before Round Two

What is the constraint, written as a product? Not "conversion" but "work available × share taken up."

What lift did we predict, per candidate, before the run? If it isn't in state on a checkpoint that predates the acting step, it doesn't count.

What is the smallest change this design could see? If the prediction is below the floor, change the lever, not the sample size.

A framework follows:

• Predicted lift below the resolution floor: do not run.
• Predicted lift above the floor, null result, delivery path healthy: investigate the graph.
• Predicted lift above the floor, null result, delivery path degraded: wrong constraint. Return to the product.
• Any result with no forecast in state: you have an anecdote with a dashboard.

The Monday case study applies the discipline properly — evaluation built into the development cycle from the start.

The Second Null Is the Expensive One

The pilot did its job. It established, before the second cheque, that the intervention had no path to the number, that the channel could not reach the people who mattered, and that the loss being tracked was not the loss occurring.

The expensive mistake is the second pilot, funded to confirm the first, built with the same lever and the same floor. A design that cannot resolve the effect it is chasing will report a null for a real effect — and it will report it with a confidence interval that looks like an answer.

So before round two: put the forecast in state, put the resolution floor beside it, and let the checkpointer hold both. Thinking in LangGraph says data you cannot reconstruct later belongs in state. What you expected is the most quietly-rewritten fact in any pilot. Write it down before the node runs.