A benchmark that runs once tells you whether a system can get an invoice right. It does not tell you whether the system gets the same invoice right the second time, which is the property you need before you can hand an accounting control to software. If a client were to scan the same document twice, the content would be the same but the scan could look different. In this case, we must ensure that any processing of the new scan will give the same result. Otherwise the number posted to the ledger depends on which upload happened to land.
Our earlier invoice-reconciliation benchmark measured accuracy on a single pass, where a purpose-built Cadel workflow scored higher than frontier coding agents. This post reruns the same freight-invoice cases many times, with the response cache off, and reports how often each system reaches the same decision twice. The Cadel workflow posts the same accounting verdict on 96.7% of reruns. The coding agents range from 65.7% to 82.9%.
The absolute scores are also higher than in that first run, for two reasons. First, the coding agents and the internal models the workflow uses have both improved since then, so the newer versions score better on the same cases. Second, we optimised several of the deterministic parts of the workflow, such as the trip-level matching and the tolerance and decision logic, which lifted the scores further.
Why one run is not enough
A month-end close, an audit trail, and an approval decision all assume that identical inputs produce identical outputs. A benchmark that scores each system once cannot check that assumption. A system can score well on a single pass and still return a different result when the same package comes through again, and the score will not show it.
The task is line-haul invoice reconciliation: matching a freight invoice line by line against the operations workbook of trips actually run, the purchase order, and the tax e-invoice, then deciding whether to approve the package for payment, hold it as an exception, or reject it. It is a freight variant of the three-way match. We ran every solver on the same document packages, with the same output contract and the same automated scorer, and then ran them again: five reruns per case for the coding agents, with response caching disabled so nothing was served from memory.
What we measured
Repeatability is not one property, so we separate three. A system can pass one and fail another.
A fourth measure, responsiveness, checks the opposite. When a fact genuinely changes, such as a quantity crossing a tolerance or a matching PO going missing, the verdict should move, and it should move only in the way the change predicts. Stability on its own is not enough: a system that returned the same verdict no matter what the input said would be useless.
Results
Same invoice packages, same decision contract, one scorer, response caching off. Cost is indexed to the Cadel workflow at 1.00×.
| Contender | Accuracy | Score variance | Verdict stability | Cost / invoice | Latency p90 |
|---|---|---|---|---|---|
| Cadel workflow | 99.2 | 0.00 | 96.7% | 1.00× | 80s |
| Codex · GPT-5.6 Sol | 94.7 | 0.90 | 82.9% | 2.50× | 123s |
| Claude Code · Opus 5 | 94.1 | 0.56 | 77.1% | 3.75× | 225s |
| Codex · GPT-5.6 Luna | 93.2 | 2.03 | 68.6% | 0.57× | 104s |
| Claude Code · Sonnet 5 | 92.9 | 1.70 | 65.7% | 2.71× | 209s |
In the accuracy column the agents look competitive. In the verdict-stability column they fall well behind. Claude Code on Sonnet 5 is the clearest case among them: an accuracy of 92.9, close to the rest, sits on top of a posted decision that holds on only about two-thirds of its reruns. Running that unattended would mean re-running each case, noticing the disagreements, and resolving them by hand, which is the work the automation was supposed to remove.
Caching and deduplication were switched off
A fair objection is that a deployed workflow could look stable only because it remembers what it saw. The Cadel platform has two mechanisms that would do this: an LLM response cache, and content-hash deduplication that returns the stored result when the same document content arrives again. Either one makes a rerun identical for free.
We disabled both for the benchmark. The response cache was off, and every epoch re-ingested the documents, so the workflow read and reconciled each package from scratch on every run, under the same conditions as the coding agents. With those mechanisms removed, the Cadel workflow still posted the same accounting decision on 96.7% of reruns. In production, with deduplication and caching on, an identical resubmission returns an identical answer by construction, so 96.7% is the floor rather than the ceiling.
Why a workflow is more repeatable than an agent
The difference comes from how the two are built, not from which model is stronger. A Cadel reconciliation workflow uses a model only for the part of the task that is genuinely ambiguous, reading values off a page. The rest runs on fixed accounting logic: pairing invoice trips to operations rows, applying the tolerance bands, and turning the checks into an approve, exception, or reject decision. Those steps return the same output for the same input on every run, and they can be tested exhaustively at no cost.
A coding agent that is handed the raw files and a shell rebuilds that whole chain from scratch each time it runs. Any ordinary variation in the model, whether a different reasoning path or a different parse of a hundred-row workbook, carries through to the final decision, because nothing downstream pins the answer in place.
What instability lets through
The stakes show up in the auto-approval decision, where the workflow either pays an invoice or sends it to a human. Across the wider benchmark, a weaker model, Codex on GPT-5.4-mini, auto-approved 30 genuinely defective packages, an auto-approve precision of 31.8%. A bare GPT-5.4-mini let 8 through. The Cadel workflow let through zero. A single-run accuracy score does not surface this. It only appears when you measure what escaped across many runs. The same reasoning is what makes duplicate invoice detection worth doing: once an unstable system approves a bad invoice, the payment is hard to recover.
