Banks ask us one question about models more than any other: can an open-weights model, running inside our own perimeter, do this job as well as a hosted frontier model? The data residency argument behind the question is real. So is the accuracy argument against it, or it was. We re-ran the comparison in September 2026 on the test set we use to sign off our own releases, and the answer has changed. The model matters less than what you put in front of it.
The test set
Sixty agreements, chosen to be awkward rather than typical: 18 LMA-style facilities, 14 LSTA-style, 12 New York law indentures, 9 fund finance facilities and 7 bespoke bilaterals. Twenty-two are scans of signed execution versions. Across them we hold 1,840 fields with a human-verified value and a human-verified clause: financial covenant thresholds and their step-downs, basket amounts including grower legs, the definitions those thresholds depend on, reporting deadlines, and the basis each test is measured on.
A field scores only if both halves are right. The value has to match, and the citation has to land on the clause a reviewer would have pointed to. We call that citation-exact accuracy and it is the only number in this note. Value-only accuracy is higher for every model and tells you less, for reasons the faithfulness note sets out.
Four classes, two conditions
We group models by what they cost to run, not by vendor: hosted frontier models, taking the best of three; open weights in the 70B dense class on two GPUs; an open 30B mixture-of-experts on one GPU; and an 8B model that fits on a laptop. Vendor names are withheld on purpose. Versions move monthly, and the point of the exercise is the shape of the result, which has held across three runs this year.
Each class runs twice. In the first condition the model gets the document and the field list, a well-tuned prompt and a long context window, and finds its own way. In the second it gets what our production pipeline gives it: a parsed section tree, an index of defined terms with their spans, the cross-reference graph, and for each field a shortlist of candidate spans found deterministically. The job shrinks from find-and-read to choose-and-read, and the answer must cite a span identifier the validator can check.
The result
| Class | Model alone | With pre-pass | Lift |
|---|---|---|---|
| Frontier, hosted | 93.1% | 97.4% | +4.3 |
| Open, 70B class | 88.2% | 96.1% | +7.9 |
| Open, 30B mixture | 86.9% | 95.2% | +8.3 |
| Open, 8B class | 71.3% | 88.4% | +17.1 |
The top of the second column is our production first pass: 97.4 percent of fields right and cited to the right clause before anyone looks at them. The routed second read takes the same set to 99.7.
Three things stand out. The frontier still leads on raw reading, by five points over the best open model. Almost all of that lead sits in two places: resolving a defined term through more than two hops, and citing the operative clause rather than a recital or a schedule that repeats the number. Where a field is a threshold in a covenant section with no indirection, all four classes land within two points of each other in either condition. And the pre-pass helps everyone, but it helps the open models most, because it does for them the thing they are worst at.
The gap between open and frontier is not a reading gap. It is a navigation gap, and navigation is a parsing problem, not a language problem.
Where the small model still fails
Nearly twelve percent of fields wrong is still too many, so we read what the 8B model gets wrong even with candidates in front of it. Two thirds are grower baskets, where the answer is the greater of a fixed amount and a percentage of a defined measure and the model returns only the fixed leg. The rest cluster on step-down schedules in tables, where it reads the right row and the wrong column. Both are shapes a validator can flag, so in production those field types are routed away from small models rather than left to fail.
What we do with it
We route by field type and by deployment. Hosted frontier models are the default for the hardest fields where a client permits them. For clients who need everything inside their own estate, the 70B class with the full pre-pass sits a point and a half behind, and the review budget in the operations note closes that gap to the same 99.7. The 8B class is a triage tool, good for classifying pages and proposing candidates, not for the final answer. Either path runs end to end, from a document landing to every affected test recomputed, inside nine seconds.
The lesson shapes the whole pipeline. Spend the engineering on what happens before the model. That work is deterministic, testable and cheap, and it is where the accuracy lives.