We asked seven AI solvers — frontier coding agents, an agent with a reconciliation skill, and a purpose-built workflow — to reconcile the same 35 freight invoices against their purchase orders, operations workbooks, and tax e-invoices, and graded every answer with one shared scorer. The purpose-built workflow scored an aggregate 97.0 / 100. The best frontier coding agent came within three points of accuracy but cost roughly five times as much per case.
The benchmark gap in accounting
There is a large number of coding benchmark datasets today. Models and agent harnesses are evaluated against them continuously, and the results give users some confidence about the ability of these agents on general coding tasks.
For finance and accounting, few reasonable benchmarks exist, and they target specific tasks: FinQA[1] and DocFinQA[2] for numerical reasoning over financial statements, FinVerBench[3] for financial statement verification, among others. Most of these are finance benchmarks and not representative of real-world accounting work. A recent benchmark called AccountingBench[4] covers closing-the-books tasks. That still leaves a gap: there is no benchmark for accounting workflow tasks.
So we are setting out to build one, based on real-world use cases. It will let us evaluate different agent harnesses and LLM models on their ability to automate accounting workflows. The first task we cover is line-haul invoice reconciliation.
The problem: line-haul invoice reconciliation
A line-haul invoice is a freight or logistics invoice issued by a supplier for the transportation leg of a shipment. It includes charges like base freight, fuel surcharge, distance-based overage charges, and container-based charges. The invoice does not contain the operational details of the transportation; those are covered in a separate operational workbook. The finance team needs to reconcile the invoice against this workbook — a freight-specific cousin of the three-way match — and also against purchase orders, supplier agreements, and tax statements, before approving the invoice for payment.
An incomplete or incorrect reconciliation during the financial-controls process can result in overpayments to vendors, or in fraud going undetected. A slow reconciliation process leads to late-payment penalties.
Benchmark design
Task
We define the task as reconciling a line-haul invoice document against an operational workbook spreadsheet, a purchase order document, and a tax e-invoice. The reconciliation result must include a matching status, an approved amount, a variance, and evidence for each match or mismatch.
Data
The case corpus includes 35 synthetic fixtures inspired by real-world use cases; no client data is used anywhere. Complexity comes from the number of line items, n-way reconciliation mismatches, many-to-one and one-to-many mappings between invoice lines and operational trips, scanned or image-only documents, and packages that do not match at all.
The cases split into three difficulty tiers. The 6 easy cases are short invoices with clean one-to-one matches. The 14 medium cases introduce duplicate or missing trips and rate-card or tax mismatches. The 15 hard cases are where the messy real world shows up: hundred-row workbooks, split and merged line items, scanned documents, and packages that do not reconcile at all.
Metrics
How close the extracted field values are to the expected values.
Exact-match scoring of the reconciliation result itself: the line-level match links between invoice lines and operational trips, the approved amount and variance, the missing / duplicate / rate-mismatch findings, and the decision status with its exception type.
A weighted average across the sub-tasks of field extraction, match and reconciliation status, and evidence collection. In plain terms: did the solver pull the right numbers out of the documents, reach the right decision (matched, exception, or no match) with the right variance amount, and point to the right evidence for each mismatch. A score of 100 means the answer matches the expected result on every graded field.
Solvers
Claude Code, Codex, and OpenCode on frontier and smaller models, with access to basic tools (bash, grep, file reads). Each agent is handed the raw files and left to work out the answer.
Codex handed the reconciliation to a reconcile skill — a reusable skill that handles general reconciliation tasks across a large number of files and different data formats, and plugs into an existing coding agent. The benchmarked run is Codex on gpt-5.4-mini, the same agent and model as the no-skill Codex row, so the two measure the same setup with and without the skill.
A purpose-built workflow for this task: an orchestration of Cadel platform primitives that internally use a combination of LLM agents and deterministic logic for targeted functions — classification, extraction, validation, reconciliation.
Scorer and instrumentation
All 35 cases are graded by the same automated scorer against known-correct answers, so every accuracy number in this post is directly comparable. Metrics are calculated per case and then aggregated. End-to-end wall-clock time and total LLM cost are also instrumented for every run; we report cost relative to the Cadel workflow (1×) rather than in absolute dollars. We use Inspect[5] to run the evals uniformly across all solvers.
Results
| Solver | Overall | Recon | Easy | Med | Hard | Rel. cost | Latencya |
|---|---|---|---|---|---|---|---|
| Cadel Workflow | 97.0 | 95.9 | 100 | 99.1 | 93.8 | 1.0× | 55s |
| Claude Code · opus-4-8 | 93.9 | 85.4 | 97.7 | 96.7 | 89.7 | 4.6× | 161s |
| Codex · gpt-5.5 | 93.7 | 85.9 | 97.7 | 96.4 | 89.5 | 3.7× | 100s |
| OpenCode · glm-5.2 | 82.8 | 76.5 | 82.5 | 83.8 | 82.0 | 1.5× | 228s |
| Codex · Reconcile Skill v1 | 77.6 | 72.8 | 97.7 | 81.1 | 66.3 | 1.3× | 114s |
| Codex · gpt-5.4-mini | 64.3 | 50.9 | 75.4 | 72.5 | 52.3 | 0.4× | 66s |
| Claude Code · sonnet-4-6 | 10.8 | 9.4 | 0 | 13.7 | 12.4 | 3.5× | 230s |
Observations
1. Easy cases are near-perfect for everyone; the hard cases separate the solvers
On the easy cases all the solvers land between 97.7 and 100. The hard cases are where they are different. The Cadel workflow holds at 93.8, the frontier coding agents come in around 89 to 90, and the smaller models fall somewhere in the 52 to 66 range. The frontier agents hold up well here, considering all they get is the raw files and a shell to work in.
2. The agents get the reconciliation answer right but not the corresponding evidence accurately
The frontier agents match the workflow on overall accuracy, but they fall behind on reconciliation accuracy — 85 against 96 in Table 1. Almost all of that gap is the line-by-line evidence. A reconciliation is more than finding the matched-vs-unmatched call and variances. You also have to say how each invoice line matches up: which operational trip it goes with, and whether it is a straight one-to-one, a split, a merge, or a partial. They agents get the totals and the decision right even for hard cases, but they wire the links wrong between matched and unmatched trips. The workflow gets the evidence right with much higher accuracy. That is the part an audit actually needs.
3. The reconcile skill boosts a small model a lot, but the hundred-row cases still need work
Running the same small model — Codex on gpt-5.4-mini — with and without the reconcile skill isolates what the skill adds, and it adds a lot. Overall accuracy jumps from 64.3 to 77.6, and reconciliation accuracy from 50.9 to 72.8. On the easy cases the skill pulls the small model all the way up to 97.7, level with the frontier agents. Where it still falls short is the hard tier — the cases with hundred-row operational workbooks, where each invoice line has to be traced across hundreds of trips. There the skill reaches only 66.3, up from 52.3 without it but well below the workflow's 93.8. A single skill call does the reconciliation in one pass and loses the iterative cross-checking the full workflow does. This is v1 of the skill, and the hundred-row cases are exactly where it needs the most improvement — that is what we are working on next.
4. glm-5.2 is slow, and it doesn't always return the format we asked for
glm-5.2 takes the most turns to get to an answer — about 16 a case, against 8.3 for Codex on gpt-5.5. It is the slowest in the set at around 228 seconds a case, and works out to roughly four times the cost of gpt-5.4-mini. It is also loose about the output format. On 4 of the 35 cases it worked the reconciliation out fine but wrote it up as prose or a markdown table (once it returned nothing at all) instead of the JSON we asked for, and the scorer gives those a zero. Same problem sonnet-4-6 has further down, milder here.
5. glm-5.2 has no native vision — so it needs a document-extraction toolchain*
glm-5.2 doesn't support images or PDFs directly. The other agents read the document as it is, but glm needs the text pulled out first. So we gave its sandbox pdftotext for the digital PDFs and tesseract for the scanned ones, and it extracts the text before it starts reasoning. Once it can actually see what is in the documents, its scores on the scanned cases come up to about where the rest of the tier sits. The glm numbers in the table are from that setup. Without the extraction tools it lands in the mid-70s, and most of that is documents it couldn't read, not the reconciliation.
6. The sonnet-4-6 outlier
Claude Code on sonnet-4-6 scores 10.8, far below everything else. Looking at the transcripts, the model actually solves the problems reasonably well — it just does not follow the output-format instructions. On 31 of 35 cases it produced a markdown report instead of the required JSON, which the scorer marks as zero. When we re-parsed the recoverable outputs, several cases scored at or near 100. We report the as-emitted number, because downstream automation can only use output in the format it expects.
The Cadel workflow and the reconcile skill
The Cadel workflow used here is a purpose-built workflow for the line-haul reconciliation task, built from the same modules behind Cadel's reconciliation product. It is an orchestration of Cadel platform primitives, which internally use a combination of LLM agents and deterministic logic for targeted functions: classification, extraction, validation, reconciliation. Apart from higher accuracy on the task, this design gives reliability and consistency in performance from run to run. Cadel workflows also automatically provide audit trails and governed data storage: every extracted figure and match decision is stored with provenance and role-based access control.
The reconcile skill used in this benchmark handles general reconciliation tasks across a large number of files and different data formats. It plugs into an existing coding agent — here we ran it with Codex — so the same skill can be reused across tasks rather than being tied to one purpose-built workflow.
Limitations
- n = 35. A single case can move a tier score by a few points. We plan to grow the corpus as the series continues.
- Synthetic fixtures. The cases are modeled on real-world use, but real client documents are messier than fixtures.
- One run per solver. Agents ran once per case with default configurations, no best-of-k. Output-format retries would likely lift the sonnet-4-6 row substantially.
- Automated scoring. One shared scorer against known-correct answers keeps the comparison fair, but scorer design choices (such as scoring the emitted output rather than the underlying reasoning) shape the numbers.
What's next
Line-haul reconciliation is the first task in this benchmark series. We are building further tasks covering other high-volume accounting workflows, on the same harness with the same instrumentation. If there is an accounting workflow you would like to see benchmarked, write to us.
References
- [1]finqasite.github.io —
FinQA: A Dataset of Numerical Reasoning over Financial Data (EMNLP 2021). Question answering over real-world earnings reports.
- [2]arxiv.org/abs/2401.06915 —
DocFinQA: A Long-Context Financial Reasoning Dataset (arXiv:2401.06915). Extends FinQA to full filings.
- [3]arxiv.org/abs/2605.29586 —
FinVerBench: a benchmark for financial statement verification (arXiv:2605.29586).
- [4]accounting.penrose.com —
AccountingBench: a long-horizon benchmark where models close the books of a real business (Penrose).
- [5]inspect.aisi.org.uk —
Inspect: an open-source framework for large language model evaluations, from the UK AI Security Institute. Used as the evaluation harness for every run in this post.