The obvious way to build document extraction with a language model is to hand it the document and ask. It works well enough in a demo to be dangerous. The model reads a leverage threshold from the recitals, cites the covenant section, and returns the answer with the fluency of someone who has done it a thousand times. Nothing downstream can tell that the citation is decorative. We built it that way first, in 2024. This note is about why we took it apart, and what the order of the two stages that replaced it buys: a first pass at 97.4 percent citation-exact, 99.7 after review, and every affected test recomputed inside nine seconds of new data.
Two stages, fixed order
The pipeline has a parser and a model, and the parser always runs first. The parser has no model in it. Given the same document it produces the same output, and a regression suite of several hundred agreements says so on every release. The model is only allowed one kind of work.
Stage one builds the map
The parser produces four things. A section tree from the table of contents and the numbering, with every heading and its page range. A definitions index: every defined term, the span that defines it, and every span that uses it. A cross-reference graph, so that "subject to Section 7.02(b)" is an edge rather than a string, and "as defined in the Intercreditor Agreement" is an edge that leaves the document and is marked as such. And a typed token layer: every number, percentage, currency amount, date and ratio, with its position on the page.
From those four it derives, for each field, a shortlist of candidate spans. For a leverage threshold: the spans in the financial covenants section that contain a ratio token, any span that section cross-references, and the definition of the ratio's own defined term. Usually three to eight candidates, each with an identifier and a bounding box.
Stage two chooses and reads
The model is given the field, the candidates, and the definitions the candidates depend on. Its job is to choose the span that answers the question and read the value out of it. Its answer is a value and a span identifier. An answer without an identifier is not an answer. An identifier that was not in the candidate list is not an answer either. Hallucinated section numbers stopped existing the day this rule went in, because a span identifier is not something a model can invent.
The validator between them
Before anything reaches the register, one check runs that costs nothing: does the cited span contain the value? The parser already knows every number in the document and where it sits, so this is a lookup. If the model says 3.50x and cites span 412, span 412 must contain a token that normalises to 3.50. If it does not, the answer goes back to the model once with the mismatch stated, and if it fails again, to a person.
That check removed the single most damaging failure we had: a correct-looking value attached to a plausible-looking clause that did not say it. The faithfulness note measures it. Before the validator, 4.6 percent of fields had the right value and the wrong clause. Now it is 1.1, inside a first pass of 97.4.
A citation is only worth having if something other than the model can check it. So the parser has to know where every number is before the model is asked what any of them mean.
Why not a better prompt
We tried. Long context, chain-of-thought, asking the model to quote before answering, asking it to grade its own confidence. Each helped a little on the average and none fixed the tail, because they all leave the model responsible for navigation. Navigation is structural. The document has a numbering scheme, a definitions section and a set of cross-references, and those are facts a parser can extract with certainty. Handing them to the model as facts, rather than hoping it infers them, is the whole difference. The benchmark puts numbers on it: the same pre-pass lifts every model class, and lifts the weaker ones most.
Nine seconds
The order also sets the speed. Because the parser is deterministic it is fast and it caches: the section tree and the definitions index are built once per document version. When a compliance certificate or a NAV notice lands, only the candidate search, the model call on the affected fields, the validator and the test engine run. From the data arriving to every affected covenant test recomputed, extraction included, the path completes inside nine seconds. The register never shows a number that is newer in a mailbox than on the page.
What the parser cannot do
It cannot tell you that "the greater of £50 million and 35% of Consolidated EBITDA" is a grower basket with two legs, or that a step-down schedule in a table applies to the test on the row above. Those are reading tasks, and the model is good at them. The split is clean: structure from the parser, meaning from the model, and a check between them that neither can fake. Confidence is calibrated per field because the validator gives a ground truth for part of every answer. Review is routed by that confidence. And every number in the register has a page box behind it that a person can click, because the parser put it there before the model was asked.