We asked nine AI solvers — frontier coding agents, an agent with a reconciliation skill, and a purpose-built workflow — to reconcile the same 35 freight invoices against their purchase orders, operations workbooks, and tax e-invoices, and graded every answer with one shared scorer. The purpose-built workflow scored 97.0 / 100 at $0.11 per case — one case being the full document package. The best frontier coding agent came within three points of accuracy, at nearly five times the cost.
The benchmark gap in accounting
There is a large number of coding benchmark datasets today. Models and agent harnesses are evaluated against them continuously, and the results give users some confidence about the ability of these agents on general coding tasks.
For finance and accounting, few reasonable benchmarks exist, and they target specific tasks: FinQA[1] and DocFinQA[2] for numerical reasoning over financial statements, FinVerBench[3] for financial statement verification, among others. Most of these are finance benchmarks and not representative of real-world accounting work. A recent benchmark called AccountingBench[4] covers closing-the-books tasks. That still leaves a gap: there is no benchmark for accounting workflow tasks.
So we are setting out to build one, based on real-world use cases. It will let us evaluate different agent harnesses and LLM models on their ability to automate accounting workflows. The first task we cover is line-haul invoice reconciliation.
The problem: line-haul invoice reconciliation
A line-haul invoice is a freight or logistics invoice issued by a supplier for the transportation leg of a shipment. It includes charges like base freight, fuel surcharge, distance-based overage charges, and container-based charges. The invoice does not contain the operational details of the transportation; those are covered in a separate operational workbook. The finance team needs to reconcile the invoice against this workbook — a freight-specific cousin of the three-way match — and also against purchase orders, supplier agreements, and tax statements, before approving the invoice for payment.
An incomplete or incorrect reconciliation during the financial-controls process can result in overpayments to vendors, or in fraud going undetected. A slow reconciliation process leads to late-payment penalties.
Benchmark design
Task
We define the task as reconciling a line-haul invoice document against an operational workbook spreadsheet, a purchase order document, and a tax e-invoice. The reconciliation result must include a matching status, an approved amount, a variance, and evidence for each match or mismatch.
Data
The case corpus includes 35 synthetic fixtures inspired by real-world use cases; no client data is used anywhere. Complexity comes from the number of line items, n-way reconciliation mismatches, many-to-one and one-to-many mappings between invoice lines and operational trips, scanned or image-only documents, and packages that do not match at all.
The cases split into three difficulty tiers. The 6 easy cases are short invoices with clean one-to-one matches. The 14 medium cases introduce duplicate or missing trips and rate-card or tax mismatches. The 15 hard cases are where the messy real world shows up: hundred-row workbooks, split and merged line items, scanned documents, and packages that do not reconcile at all.
Metrics
How close the extracted field values are to the expected values.
Exact-match scoring of the reconciliation result itself: the line-level match links between invoice lines and operational trips, the approved amount and variance, the missing / duplicate / rate-mismatch findings, and the decision status with its exception type.
A weighted average across the sub-tasks of field extraction, match and reconciliation status, and evidence collection. In plain terms: did the solver pull the right numbers out of the documents, reach the right decision (matched, exception, or no match) with the right variance amount, and point to the right evidence for each mismatch. A score of 100 means the answer matches the expected result on every graded field.
Solvers
Claude Code, Codex, and OpenCode on frontier and smaller models, with access to basic tools (bash, grep, file reads). Each agent is handed the raw files and left to work out the answer.
A coding agent that hands the reconciliation to the Cadel reconcile skill. Cadel skills are Cadel modules exposed through the Cadel APIs and CLI, and can be used with any existing coding agent. This one is optimized for handling different data formats and reconciling across a large number of files.
A purpose-built workflow for this task: an orchestration of Cadel platform primitives that internally use a combination of LLM agents and deterministic logic for targeted functions — classification, extraction, validation, reconciliation.
Scorer and instrumentation
All 35 cases are graded by the same automated scorer against known-correct answers, so every accuracy number in this post is directly comparable. Metrics are calculated per case and then aggregated. End-to-end wall-clock time and total LLM cost are also instrumented for every run. We use Inspect[5] to run the evals uniformly across all solvers.
Results
| Solver | Overall | Recon | Easy | Med | Hard | $/case | Latencya |
|---|---|---|---|---|---|---|---|
| Cadel Workflow | 97.0 | 95.9 | 100 | 99.1 | 93.8 | $0.114 | 55s |
| Claude Code · opus-4-8 | 93.9 | 85.4 | 97.7 | 96.7 | 89.7 | $0.530 | 161s |
| Codex · gpt-5.5 | 93.7 | 85.9 | 97.7 | 96.4 | 89.5 | $0.424 | 100s |
| OpenCode · opus-4-8 | 93.6 | 85.4 | 97.7 | 95.9 | 89.9 | $0.532 | 161s |
| OpenCode · gpt-5.5 | 93.3 | 85.1 | 97.7 | 96.0 | 89.0 | $0.327 | 140s |
| OpenCode · Cadel Reconcile Skill | 77.6 | 72.8 | 97.7 | 81.1 | 66.3 | $0.145 | 114s |
| OpenCode · glm-5.2 | 75.6 | 70.8 | 80.4 | 68.5 | 80.3 | $0.174 | 243s |
| Codex · gpt-5.4-mini | 64.3 | 50.9 | 75.4 | 72.5 | 52.3 | $0.043 | 66s |
| Claude Code · sonnet-4-6 | 10.8 | 9.4 | 0 | 13.7 | 12.4 | $0.397 | 230s |
Observations
1. Easy cases are near-perfect for everyone; hard cases separate the solvers
On easy cases, all solvers get near-perfect accuracy (97.7–100). The separation happens on the 15 hard cases: the Cadel workflow holds at 93.8, the frontier coding agents land at 89–90, and the smaller models drop to 52–66. The frontier agents do well here for systems handed nothing but raw files and bash.
2. Our one-shot reconcile skill drops off on hard cases
The reconcile-skill condition matches the frontier agents on easy cases (97.7) but drops to 66.3 on the hard tier. A single skill call loses the iterative cross-checking that both the full workflow and the agents do. We are working on closing this gap.
3. glm-5.2 takes far more turns to reach the same result
glm-5.2 averaged 18.1 turns per case, against 8.3 for Codex on gpt-5.5, and consumed 15.7M tokens across the run against 6.4M. So even though glm-5.2's per-token prices are low, its per-case cost comes out at $0.174 — four times gpt-5.4-mini's — and at 243s per case it is the slowest solver in the matrix.
4. The harness affects cost, not just the model
Running the same model through different agent harnesses costs different amounts. On gpt-5.5, Codex came out at $0.424 per case against OpenCode's $0.327: OpenCode sends a smaller system-prompt context to the model on every turn, so the same work costs less. The effect is not universal, though — on opus-4-8, Claude Code and OpenCode landed within half a cent of each other ($0.530 vs $0.532).
5. glm-5.2 has no native vision support
glm-5.2 does not support image understanding natively. Several of the harder cases involve scanned or image-only documents, and this becomes a bottleneck for the model there.
6. The sonnet-4-6 outlier
Claude Code on sonnet-4-6 scores 10.8, far below everything else. Looking at the transcripts, the model actually solves the problems reasonably well — it just does not follow the output-format instructions. On 31 of 35 cases it produced a markdown report instead of the required JSON, which the scorer marks as zero. When we re-parsed the recoverable outputs, several cases scored at or near 100. We report the as-emitted number, since downstream automation can only consume output in the specified format.
The Cadel workflow and the reconcile skill
The Cadel workflow used here is a purpose-built workflow for the line-haul reconciliation task, built from the same modules behind Cadel's reconciliation product. It is an orchestration of Cadel platform primitives, which internally use a combination of LLM agents and deterministic logic for targeted functions: classification, extraction, validation, reconciliation. Apart from higher accuracy on the task, this design gives reliability and consistency in performance from run to run. Cadel workflows also automatically provide audit trails and governed data storage: every extracted figure and match decision is stored with provenance and role-based access control.
Cadel skills are Cadel modules exposed through the Cadel APIs and the Cadel CLI tool, and can be used with any existing coding agent. The reconciliation skill used in this benchmark is optimized for handling different data formats and doing general reconciliation tasks across a large number of files.
Limitations
- n = 35. A single case can move a tier score by a few points. We plan to grow the corpus as the series continues.
- Synthetic fixtures. The cases are modeled on real-world use, but real client documents are messier than fixtures.
- One run per solver. Agents ran once per case with default configurations, no best-of-k. Output-format retries would likely lift the sonnet-4-6 row substantially.
- Automated scoring. One shared scorer against known-correct answers keeps the comparison fair, but scorer design choices (such as scoring the emitted output rather than the underlying reasoning) shape the numbers.
What's next
Line-haul reconciliation is the first task in this benchmark series. We are building further tasks covering other high-volume accounting workflows, on the same harness with the same instrumentation. If there is an accounting workflow you would like to see benchmarked, write to us.
References
- [1]finqasite.github.io —
FinQA: A Dataset of Numerical Reasoning over Financial Data (EMNLP 2021). Question answering over real-world earnings reports.
- [2]arxiv.org/abs/2401.06915 —
DocFinQA: A Long-Context Financial Reasoning Dataset (arXiv:2401.06915). Extends FinQA to full filings.
- [3]arxiv.org/abs/2605.29586 —
FinVerBench: a benchmark for financial statement verification (arXiv:2605.29586).
- [4]accounting.penrose.com —
AccountingBench: a long-horizon benchmark where models close the books of a real business (Penrose).
- [5]inspect.aisi.org.uk —
Inspect: an open-source framework for large language model evaluations, from the UK AI Security Institute. Used as the evaluation harness for every run in this post.