Testing LangGraph Graphs With Planted Defects: Tests That Must Fail

A guard test is proven only once you have watched it fail on a defect you planted on purpose — and the plant stays, so it fails again on every run. Every guard worth keeping asserts an absence: no node writes, every model node is guarded, every declared file ships. Absence has a pass path that has nothing to do with your code. The scan looked in the wrong directory, the pattern never matched, the node list it walked was empty.

LangGraph in production, a 10-part series. ← Part 9

A Guard That Asserts an Absence Can Pass Without Reading Your Code

A warehouse graph, a few nodes reading picks and shipments through a read() helper, and one plant — planted_write, calling read("update shipments set late = true"). The first checker knows one spelling:

def naive_writes(compiled):
    return [n for n, fn in node_functions(compiled).items()
            if re.search(r"\bINSERT\b", inspect.getsource(fn))]

naive  clean: pass  planted: pass

Both green. The regex is case-sensitive, so a lower-case update is invisible; it never parses, so an assembled statement is invisible too. The remedy isn't a better regex. It's a bad input kept permanently beside the guard, in the same parametrised test, which asserts pass on the clean graph and fail on its twin.

The Scan Has to Prove It Inspected Something, Node by Node

The second checker parses instead of matching — ast "helps Python applications to process trees of the Python abstract syntax grammar" (Python docs) — and walks each node for read calls. A total offender count still proves nothing, because one clean node can hide a node nobody inspected. So it counts inspections per node: zero read() calls fails unless the node sits in an explicit no_io set. A graph with nothing to scan fails as scanned nothing. That rule earns its keep on wrapped nodes: a decorator without functools.wraps hides the real function from inspect.unwrap, so the scan reports logged_write: no read() seen — absence is the finding, not the pass.

Enumerate From the Registry the Graph Was Built From

A hand-kept node list falls exactly one node behind the graph, and the node it misses is the writer. Take the list from the name-to-function dict the graph is built from; a new node is then covered the moment it exists. Cover async def too. LangGraph documents that "nodes are Python functions (either synchronous or asynchronous)" (graph API) and stores async ones as afunc, not func. So the read is func or afunc. Reaching into compiled.builder.nodes works in the installed version, but there is no public node-listing API; treat it as the fallback and your own registry as the contract.

The Database Beats the Parser, and It Is Not Close

Loading diagram…

"Every query starts with SELECT or WITH" is a demo rule, and two statements walk past it: WITH u AS (UPDATE shipments SET late = true RETURNING 1) SELECT count(*) FROM u and SELECT setval('ship_seq', 1). Both parse. Both start with an allowed keyword.

prefix rule: pass | read-only transaction: rejected
prefix rule: pass | read-only transaction: rejected

The first is a reporting query that mutates every late flag on the shipments table, and every layer above waves it through. The guard that cannot be fooled by spelling is the engine: run readers inside SET TRANSACTION READ ONLY, or as a role with no write grants, and Postgres refuses both statements before your rule gets a vote.

The Objection Is Right About One Layer and Wrong About the Rest

The case against all of this: permanent plants are upkeep that mostly tests itself, and where an invariant really matters you should enforce it in the database and delete the scan. That is correct about the layer with the best evidence in the example, and it is where a team should spend first. It doesn't carry the rest. The database rejects a statement; it cannot tell you which node emitted it, that a model node skipped its checker, or that a declared file lost its ignore exception. Those are properties of the source tree, and the only evidence a source-tree guard works is watching it name a plant. What the plants cost in CI time and upkeep is real; it is a trade, not a finding.

Alternatives, and What Each Costs

An off-the-shelf mutation runner buys this discipline across a whole package, at a full suite rerun per mutant and a tranche of equivalent mutants you triage by hand. It is worth it when the code is broad and mostly pure — not when the thing to verify is a scan of your own graph. A one-time red run plus a review checkbox costs nothing and expires the moment the guard or the graph changes. Assertions on returned state catch real behaviour but cannot see a node that bypasses the helper, and over a non-deterministic model node they are the flakiest thing in the suite. pytest.mark.parametrize is enough to push clean and planted through one test body: "The builtin pytest.mark.parametrize decorator enables parametrization of arguments for a test function" (pytest docs).

Practical Takeaways

• Keep the plant beside the guard it proves, parametrised with the clean input.
• Count inspections per node, declare no_io explicitly, fail an empty scan.
• Enumerate from your own registry; read afunc as well as func.
• Parse before you grep, and push any invariant that can be a permission into the database.

Choose the database when the invariant is expressible as a grant. Keep a plant when the guard reads your own source tree, because that is the only class of guard where a plant is the sole evidence. Skip both when the guard is about to be replaced. A check you have never seen go red isn't a weak test. It is an empty slot where a test should be.