Last month a bank sent us a credit agreement to see what ExactCov would make of it. It was 892 pages: the amended and restated facility, four amendments, an incremental joinder, the intercreditor and a security agreement, all in one scanned PDF with the schedules at the back and the pricing grid replaced twice. The kind of document that gets read properly once, at closing, by the lawyers who drafted it.
We ran it in the afternoon. This post is about what came out, what the AI got wrong, and how we knew.
What came out
| Item | Count |
|---|---|
| Pages read | 892 |
| Pages holding covenant provisions | 187 to 341, plus 62 pages of definitions |
| Covenants in the register | 47 |
| Ratio and threshold tests | 61 |
| Baskets, with their capacity and grower components | 112 |
| Carve-outs and exceptions linked to a covenant | 203 |
| Defined terms with a plain-English summary | 418 |
| Citations, each a verbatim quote on a named page | 1,014 |
| Citations machine-verified against the page | 1,003 |
| Items routed to a human reviewer | 11 |
| Wall-clock time | 27 minutes |
The 47 covenants are the ones you would expect from a large-cap leveraged facility: two maintenance tests, a long negative covenant package with indebtedness, liens, restricted payments, investments, asset sales and affiliate transactions, the reporting and notice obligations, and the events of default that hang off them. The 61 tests are the ratios and thresholds inside those covenants, including the ones that only exist to unlock a basket. The 112 baskets are where the document earns its length, because most of them are the greater of a fixed amount and a percentage of EBITDA, and the percentage points at a definition that runs to four pages.
What the AI got wrong
Eleven items did not survive our checks and went to a person. Three are worth describing, because they are the shape of mistake anyone reading a document like this makes, human or not.
- A superseded pricing grid. The third amendment replaced the margin grid in full. The first pass extracted both grids and could not say which was live. The reviewer marked the original as superseded with a citation to the amendment, and the register now shows the grid in force and the one it replaced.
- A basket defined by reference to another agreement. One investment basket was sized by reference to a capacity in the intercreditor. The extraction captured the cross-reference but not the number, and said so. That is the right outcome. A confident wrong number is the failure mode we design against; an honest gap is a task for a reviewer.
- An EBITDA add-back cap. The add-back for cost savings was capped at a percentage of EBITDA "prior to giving effect to this clause". The first pass read the cap as applying after. A second pass over the same passage corrected it, and both readings are in the audit trail with the quote that decided between them.
Everything else passed the check that matters: the quote the model gave for each figure is on the page it cited, word for word, and the number in the register appears inside that quote. A citation that fails that test never reaches the register without a person looking at it.
How we know, without saying how we do it
People ask how the models are arranged, which ones we use and what they are told. We do not publish that, and not out of coyness. It changes every quarter, it is the part of the work that took the longest to get right, and it is not the reason to trust the output. The reason to trust the output is that you can check it, one page at a time, and that we check it first.
The AI reads the document. The register is only allowed to keep what the document can be shown to say.
So the promise is not "the model is very good", although it is. The promise is that every figure in front of a credit officer carries its page and its quote, that the verification is done by machine before a human ever sees the number, and that the eleven items in 892 pages the machine could not verify were the eleven items a human read. That is a review the second line can put its name to, and it is a different thing from a summary.
The agreement is now in the register with its reporting calendar set up from the covenant compliance certificate schedule. The first certificate is due in October. The tests will run the day it arrives, against those 61 thresholds, with the 892 pages behind them.