Who Is in Your LangGraph Pilot Decides the Answer

Suppose a pilot comes back positive: the treated group beats the control group on the headline measure, the delta lands on a single slide, and the room starts talking about rollout.

LangGraph in operations, a 10-part series. ← Part 8 · Part 10 →

Then someone asks: what stopped a treated contractor from taking work a control partner had already claimed? Where contractors compete for the same jobs, the treatment did not create work. It moved work. A design that lets work move between arms reports the movement as an effect, in whichever direction it happened to run.

Who is in your pilot means who is in the sample — and then who owns the rule that put them there. Not headcount. Selection. Cohort design is a business-integrity question dressed as a sampling detail: a hand-picked or reallocation-prone sample turns a null into a headline, and the fix is a rule plus a hash, computed once, persisted, and read by every later run and every later audit.

Interference Flatters the Arm You Are Rooting For

If two contractors can see the same pool of jobs, randomisation does not isolate the treatment — it only decides who reaches the queue first. The treated arm's advantage is then partly a transfer rather than a gain, and no statistical care inside the analysis can undo that, because the contamination happened in the world before the data was collected.

The remedy is unglamorous: overlapping work that both arms can see gets held back, and the held-back slice gets counted. A withheld slice is a measured quantity — it tells you how much of the business the pilot cannot speak to — whereas an unmentioned exclusion looks identical to an exclusion that never happened.

The exclusion has to live somewhere the rest of the pipeline can see it. LangGraph is built to combine hand-coded deterministic steps with LLM-driven ones in one graph, so an eligibility filter is a predictable node rather than a judgement call made by whoever runs the analysis that week (LangGraph overview).

Ranking on a Good Stretch Guarantees the Pilot Reports a Decline

Ranking the cohort on a recent good stretch imports regression to the mean. A unit lands at the top of a noisy measure partly because the noise went its way; measure again and the noise reverts. Select both arms on a peak and both arms will probably look worse next period, and the pilot will report a decline the programme did not cause.

A Rule Plus a Hash Is a Cohort; a List Is an Anecdote

The replacement is not a better list. It is a rule: rank on a measure taken from the pre-period, exclude units that were inactive in the pre-period, pair similar units, and let a fresh random key split each pair.

Rules are also the only version of a cohort that can be an instrument rather than a snapshot. New Computer's memory-retrieval work is the clean example: queries generated from a cohort of synthetic users across a wide range of intents, prompt changes compared in an experiment view, and 50% higher recall and 40% higher precision measured against their previous baseline implementation (New Computer case study). The sample was a generated cohort rather than a hand-kept list; a cohort defined by a rule can be regenerated, which is what makes a repeat measurement possible.

LangGraph Already Has a Place to Put the Cohort

Assignment gets computed once by the rule and written to durable state, after which every consumer only reads it, by convention and by keeping the history in a durable saver. The persistence model draws the line precisely: a checkpointer holds a single thread's graph state keyed by thread_id, a Store holds application-defined data shared across threads, and most applications use both (LangGraph persistence). A cohort that every run and every audit must read belongs in a pinned checkpoint thread or in a Store, never in a function that recomputes it per run — LangGraph's built-in memory keeps context over time and across sessions, exactly the property a cohort needs, because the cohort is read long after the run that produced it has finished (LangGraph).

Keep in state what cannot be reconstructed later; compute what can be derived (Thinking in LangGraph). Applied correctly, that means store the rule, the hash, and the snapshot of eligibility data the rule ran against. The rule is deterministic; the data underneath it is not. Contractor activity changes next month, so persist only the rule and you have persisted a machine for producing a different cohort on demand.

A Cohort Every Run Recomputes Will Eventually Disagree With Itself

Analysis runs early in the pilot, the audit runs long after, and if both derive the cohort from current data, they get different cohorts — activity moved, rankings moved. The audit cannot reproduce the analysis, and the argument stops being about the treatment effect and becomes about whose query was right.

LangGraph's time-travel semantics handle this better than most people expect. update_state does not roll back a thread; it creates a new checkpoint branching from the chosen point, leaving the original execution history intact, and get_state_history exposes every checkpoint (LangGraph time travel). Correct the assignment rule and you get a visible fork: the original cohort and the corrected one both exist, with the divergence point recorded.

The Decision Framework: Three Questions and One Artifact

Before the pilot result goes upward, make the team answer three questions, in this order, in writing.

1. What work can move between arms? Enumerate the overlaps. Hold them back. Report the held-back volume as a first-class number, not an appendix.
2. What selection put each unit in the cohort? If the answer references the outcome window, stop. Selection comes from the pre-period, and any ranking on recent performance gets replaced by a rule.
3. What is the design's noise floor? State the smallest effect the design could detect. If the observed delta is inside it, the result is "we could not tell."

The artifact is one object with four parts: the rule, the hash that implements it, the pinned data snapshot it ran against, and the held-back list. Persist all four. Read them, never recompute them.

Ask for the Hash

The pilot result is a claim about a rule, not a claim about a vendor, a framework, or a model. If you cannot re-derive the cohort from the rule, the hash, and the snapshot, you do not have a result. You have a recollection.

Request the cohort rule. Request the hash that implements it. Request what was held back to prevent interference between the arms, and the count of what it was worth. Anyone who ran a real pilot can produce all three in an afternoon.

LangGraph gives you the machinery to make that answer durable: a checkpointer keyed by thread, a Store shared across threads, and a fork-visible history when the rule changes. What it cannot give you is the decision to write the rule down before you look at the outcome. That decision is made by people, in a room, and it is the first number the whole pilot depends on.