There are two easy answers to how much of an AI extraction a person should check, and both are wrong. "All of it" is a second extraction done more slowly by a more expensive reader, and it removes the reason to automate. "None of it" is a bet that the accuracy on the slide was measured on documents like yours. The right answer is a budget. Ours is 15 percent of fields, chosen by the system rather than at random, and it is the distance between 97.4 percent and 99.7.
Calibration first
Every model emits a confidence of some kind, and most are useless for routing because they are not calibrated: an answer marked 90 percent is right 70 percent of the time, and one marked 60 is right 65. Routing on that is barely better than a coin.
Our field confidence is built from signals that have a ground truth behind them, not from the model's opinion of itself. Did the span validator pass on the first attempt? How many candidate spans were there, and how far apart were the top two? How deep was the definition chain? Did the digits re-read agree with the primary OCR pass? Was the page a scan? Each is a feature, the target is whether the field was right in the labelled set, and the output is a probability we recalibrate on every release. On the current set, fields marked 95 percent are right 94.6 percent of the time. That is what makes the curve in the figure possible.
The curve
Take the production first pass on our 1,840 fields. It is 97.4 percent citation-exact, which leaves 48 errors. Sort the fields by confidence, lowest first, send the bottom slice to a reviewer, and count how many of the 48 they would find.
| Fields sent | Errors caught | Errors reaching the register | Accuracy after review |
|---|---|---|---|
| 5% | 60% | 19 | 99.0% |
| 15% | 88% | 6 | 99.7% |
| 30% | 96% | 2 | 99.9% |
| 50% | 98% | 1 | 99.95% |
A random sample of 15 percent would catch 15 percent of errors. Routed by confidence, the same 15 percent catches 88, and takes the set from 97.4 to 99.7. That is the budget we run in production. The knee of the curve sits between 12 and 18 percent depending on the document mix, and past 30 the reviewer is almost entirely reading correct answers.
Setting the budget
We express the budget as reviewer time per agreement rather than as a percentage, because that is what a team can plan. At 15 percent, on a typical 30-field agreement, that is four or five fields at about 45 seconds each: under four minutes per agreement, and sixty agreements in an afternoon. The six errors that get through are not random either. Five sit at definition depth three or more, exactly where the definitions note says the difficulty lives.
Three adjustments move the budget in practice.
- Field weight. A financial covenant threshold matters more than a notice address. Routing is weighted so a covenant field at 90 percent confidence is reviewed before an address field at 80.
- Document type. Scans and amended agreements draw a larger share. Clean digital LMA templates draw a smaller one.
- Deployment. A client running the 70B open-weights class inside their own estate, per the benchmark, starts a point and a half behind the hosted frontier. The budget rises to roughly 18 percent to land on the same 99.7.
The question is not how much to review. It is whether the system can tell you which fifteen percent. If it cannot, review everything, and ask why you bought it.
What the reviewer sees
The second read is only fast if it is designed. The reviewer gets the field, the extracted value, the cited span highlighted on the page image, the two runner-up candidates, and the reason the field was routed: low validator margin, deep chain, digits disagreement. Most reviews end in under a minute, because once the span is on screen the answer is either obviously right or obviously wrong. The ones that take longer should: a grower basket with both legs to check, a step-down table with a merged column.
What happens to the rest
The 85 percent that goes straight through is neither unreviewed nor slow. The tests that depend on it are recomputed inside nine seconds of the data landing, while a routed field waits for its second read with the test flagged rather than silently stale. Every field keeps its confidence and its citation, so a credit officer opening a test six months later can see the threshold was extracted at 97 percent from page 142 and click through. And every correction a reviewer makes is a labelled example that feeds the next calibration. The budget falls over time, on the same documents, because the system learns which of its doubts were justified.