Pre-Register Your LangGraph Pilot Before the First Dashboard

Three weeks after the read-out, the meeting happens. Not the launch review — the other one. Someone from procurement, or legal, or the VP who inherited the budget, asks how the cohort was chosen. The room looks for a document that predates the number and finds a deck that postdates it. The evaluation criteria live on a single slide, written in the week the traces started looking good. The success threshold appears in the same sentence that reports clearing it.

LangGraph in operations, a 10-part series. ← Part 3 · Part 5 →

That is not a governance failure. It is a sequencing failure. The dashboard was built first, and a dashboard is an extraordinarily persuasive instrument for deciding, afterwards, what you were always measuring.

The fix is old and unglamorous. Freeze the design before the first data point: a versioned LangGraph StateGraph per arm, an evaluation suite pinned in the repository beside the graph it judges, stop rules committed while nothing is at stake, and a written sentence naming the smallest effect the design could ever detect. Then let the dashboard do the only job it is qualified for — witnessing, not judging.

A pilot that is not pre-registered is a demo with a follow-up slide. What separates a demo from a finding is a document that predates the result.

The Dashboard Is a Witness, Not a Judge

Tracing tells you what happened; LangGraph's documentation pairs the runtime with LangSmith for tracing, evaluation and deployment LangChain (n.d.). Build the tracing surface before you write the success criteria and the surface writes them for you.

My view: the evaluation set is the most load-bearing artifact in a pilot and the least governed. Everyone reviews the graph. Nobody reviews the inputs.

Freeze the Graph, Not Just the Prompt

The thing under test is a StateGraph, not a prompt — nodes, edges, routing conditions, tool bindings, a checkpointer — so moving one conditional edge changes the treatment. Pinning the prompt string while the graph stays on main is a note to self, not a freeze.

AppFolio keeps a central repository of sample cases with ideal outputs, used as both evaluations and unit tests, run in CI, with code merging only when all tests pass and evaluation thresholds are met LangChain (n.d.). The freeze is a gate a merge cannot pass. Monday embedded evaluations in development from the start, managed as version-controlled code with CI/CD, with offline evaluations before production traffic LangChain (n.d.).

The unit of the experiment is the pair — graph and eval suite — tagged together, so any result can name both versions it came from.

An Amendment Is a Branch, Not an Edit

LangGraph's time-travel documentation states that calling update_state on a prior checkpoint creates a new branch rather than rolling the thread back, and the original execution history remains intact LangChain (n.d.).

That is the amendment model, and it is worth stealing wholesale. When you learn something mid-pilot, do not revise the design document into agreement with the new understanding. Fork from the locked checkpoint: append a dated amendment, state what changed and when, and leave the original readable forever.

An appended amendment tells a reviewer exactly which numbers were produced under which rules. An edited document tells a reviewer only what you believe now—precisely what they are unwilling to take on trust.

Most tooling makes the bad version easier: overwrite the config, rerun, present the latest. LangGraph keeps run history by default when a checkpointer is attached, but a design document is not under it; the hash-pinned test and the append-only amendment log are what carry the guarantee.

Stop Rules Are Bought Before the Read

Every pilot needs at least one condition that ends an arm regardless of its numbers: a duplicate contact, an opt-out breach, something that means the arm stopped being a valid test. Committing these before the read is arithmetic, not principle. After the read, a stop rule competes with a number somebody already likes.

LangGraph gives the discipline a technical shape: the interrupt primitive pauses a graph inside a tool before the side effect executes and lets the resume value override the inputs LangChain (n.d.). That is a stop rule at the boundary.

The Smallest Detectable Effect Is the Only Line That Changes the Decision

Executives ask two questions about a pilot: did it work, and could the test ever have seen the effect it was funded to find? The second is answerable before the first data point, or not at all.

The committed line must be readable without statistics: the smallest lift the design could see was about a third of the average. The pilot can prove a large win, not a modest one.

What Survives a Change of Sponsor

Sponsors change. Budgets get reallocated, champions get promoted or leave, and the person who funded the pilot is frequently not the person defending it two quarters later.

Pre-registration is cheap, and it is the only part of the pilot that survives that handover. A pinned hash, a stop rule, a smallest detectable effect and an amendment log can be handed to a successor who never met the team and did not approve the spend — and that successor can reach a conclusion without interviewing anyone.

The New Computer case study is the cleanest public illustration of what an evaluation harness buys once built. LangSmith's experiment comparison view let the team inspect the global effect of prompt changes and identify regressed runs derived from a change. It carried them to 50% higher recall and 40% higher precision than a previous baseline implementation of dynamic memory retrieval LangChain (n.d.). The comparison view is the operative part. Without it you have a system that improved on the queries someone happened to look at.

Practical Takeaways

• Freeze the design and pin its hash. A test asserts the file's hash; amendments are appended, the original stays byte-identical.
• Pin the pair. A StateGraph version and an evaluation-suite version, tagged together.
• Split the log. Before-lock in one section, after-lock in another.
• Commit stop rules first. Duplicate contact, opt-out breach, quota exhaustion. No post-hoc netting.
• State the smallest detectable effect with its power and window.
• Make the dashboard a monitor. It reports against committed criteria.

Closing

A pre-registration request is small: send it before the first data point. Ask for the frozen design, its hash, its stop rules, and the smallest detectable effect, then the amendment log. This costs some writing and the discomfort of not editing your own intentions; it buys the ability to tell a room looking for a reason not to believe you that the rules were written down first.

The dashboard will outlive the pilot, still rendering charts for a system three owners removed, with nobody left remembering which threshold was chosen before the data and which after. That ambiguity is the default outcome, and it is not fixed by better tooling — LangSmith renders the same trace whether or not the criteria were frozen.