A large ecommerce marketplace has a vendor base in the hundreds of thousands. Every delivery produces a proof of delivery (POD), usually a scanned PDF. Before the vendor is paid, that POD has to be checked against the item received note (IRN) and the purchase order (PO), and turned into one invoice the ERP will accept.
That takes more than one extraction pass. The invoice has a header, carrying vendor, invoice number, dates and totals, and a body of line items. The ERP won't take it unless both are right. A single POD lists dozens of lines, and the largest in our test set lists 559. Each one has to match the right line on the IRN and the PO. Get one wrong and the invoice doesn't reconcile.
This is a three-way match, run at scale, on scanned documents. We gave the same 60 cases to seven systems, scored every one of them with the same scorer, and measured all of it: the header fields, every line, and what the run cost.
The workflow
We built this workflow with an ecommerce customer. It was one of the first we built on Accos, the engine that runs every one of our Cadel workflows. Each step gets the tool that fits it.
One workflow, the right tool at each step
The PO and the IRN are structured files, a spreadsheet and a CSV, so we parse them as data. The scanned POD needs a vision model to read its line items. Pairing those OCR'd lines to the clean catalog rows is the hard step, so it gets its own model call with the candidate rows in front of it. Joining to the PO and assembling the invoice is deterministic code.
The workflow runs on several models, in three configurations named by what they spend: low, medium and high cost. In the high-cost configuration, our default, a frontier vision model reads every scan, recovers lines the first pass missed, and runs the matching calls that drive reconciliation. The medium and low tiers run those same calls on smaller models. The deterministic half of the workflow, parsing the structured files, building candidate lists, merging matched lines into the final invoice, is identical in every tier.
Seven ways to do the job
The obvious question is whether you need the workflow at all. Could you hand the raw POD, IRN and PO to a strong coding agent and let it produce the invoice? We tested that, against three configurations of our own workflow.
Model versions matter and they move fast, so to be precise: the coding agents below ran on 30 July 2026, our workflow's outputs are from 26 July, and every system is scored on the same 60 cases by the same scorer.
- Our workflow in its high-cost (default), medium-cost and low-cost configurations
- GPT-5.6-sol and GPT-5.6-luna, run through the Codex CLI
- Claude Opus 5 and Claude Sonnet 5, run through Claude Code
What the numbers say
How we measured
- Data: 60 cases from a public synthetic set, no customer data. 2,629 ground-truth line items, the largest single document carrying 559 of them. Scanned, image-only PODs. Seed 20260703. This is not a synthetic-only result: the same workflow runs in production on real customer documents. The synthetic set exists so we can publish exact numbers without exposing anyone's invoices.
- One scorer for every system. The composite weights line-item F1 at 0.70, line-field accuracy at 0.20 and header accuracy at 0.10. Identifying the right set of lines is the hard part of the task, so it carries most of the weight.
- We report the mean. The composite is computed per case, giving 60 scores per system, and the chart plots each system's mean across those 60.
- Line items are matched to the ground truth by content, and scored on F1, so a system loses points for lines it invents as well as for lines it misses. An errored case counts as zero.
- The coding agents ran in their real harnesses (Codex CLI, Claude Code) in a Docker sandbox, on 30 July 2026. Each got the three raw files in its sandbox, the scanned POD as a PDF, the IRN as a CSV and the PO as a spreadsheet, and had to produce the reconciled invoice. Our rows ran the actual product workflow end to end; their outputs are from 26 July, re-scored with the same scorer. Same scorer for all seven. Harness versions: inspect_ai 0.3.241, inspect_swe 0.2.63. The Codex rows ran the GPT-5.6 models on Codex's pinned 5.5 scaffold; its 5.6 profiles are code-mode only and our harness cannot select them.
- Cost is each system's total LLM spend for all 60 cases, reported as a multiple of our high-cost configuration's spend, at API list prices as of 31 July 2026.
Why Cadel costs less
Look at the two Claude Code runs. Sonnet 5 is the cheaper model per token, and its run cost more, because it spent 2.2 times the output tokens doing the same job. On this workload the bill is driven by how many tokens a system spends.
Controlling that is what the structure of our AP automation workflow is for. The structured files are parsed as data, the matching step works from a short candidate list, and the only document a model reads in full is the scan. A general agent re-reads the raw documents in its loop on every run, and that difference accounts for most of the gap on the cost axis.
One agent does reach our price range: Codex on GPT-5.6-luna, at 0.14× after OpenAI's 30 July price cut. That puts it between our low tier at 0.09× and our medium tier at 0.16×. It is also the lowest-scoring system in the field, with a mean of 90.2 against the medium tier's 94.1.
What comes next
This benchmark answers a narrow question: how accurately and cost-effectively can each system turn the same 60 cases into reconciled invoices? Production finance workflows have other requirements. We are extending the benchmark in three directions.
Repeatability and reliability. Finance teams need the same inputs treated consistently across runs, predictable completion, and failures that are visible and recoverable. We are rerunning the benchmark to measure score variance, completion rates, and tail failures. We will publish those results in a follow-up.
Model independence. Changes in model access, pricing and provider terms have made one risk concrete: a production workflow should not depend on a single closed-model provider. Cadel supports open-weights models alongside hosted proprietary models. We are running the same benchmark against open-weights models and will publish the results soon.
Security and data control. Invoices, purchase orders and delivery documents contain sensitive commercial and personal data. Cadel supports deployment within a customer's VPC, allowing enterprises to align model access, data residency, retention and audit controls with their own requirements. Future benchmark reports will make the deployment boundary explicit: where each model runs, what data crosses that boundary, and which retention controls apply.
We will report each of these as its own dimension, so finance teams can see the trade-offs clearly.
