We took the proof-of-delivery workflow described in our POD processing benchmark, replaced the closed frontier models inside it with a combination of open-weight models, and re-ran the same 60 cases through the same scorer. The open-weight configuration scored 95.0 against 94.6 for the identical workflow on closed models, at 0.98× the cost. It is also roughly twice as slow.
Why this question matters now
Every finance team we talk to eventually asks some version of the same question: does our data have to leave our infrastructure for this to work? Until recently the honest answer involved a trade, because the models good enough to read a smudged delivery note reliably were closed ones, hosted by a small number of vendors and reached over the public internet.
Open-weight models make that a different question. Their weights can be downloaded and served on hardware the customer controls. What matters after that is how much accuracy, cost and speed a team gives up by switching to them.
We were able to check that directly. The workflow was already in production and the benchmark was already published, so the models were the only new variable.
What we changed, and what we did not
The models were the only thing we changed. The workflow is the same one that runs in our AP automation product: documents in, a reconciled invoice out, with a deterministic engine doing the matching and the arithmetic. The 60 cases are the same scanned proof-of-delivery, receipt-note and purchase-order sets. The scorer is the same composite, weighting line-item F1 at 0.70, line-field accuracy at 0.20 and header accuracy at 0.10.
Inside the workflow, the model roles were repointed to a combination of open-weight models, including Kimi K3 and GLM. Which model serves which step is an implementation detail that changes as models improve, and we are not treating it as the result here. What makes the swap possible at all is that everything downstream of the reading step is fixed code, and that code does not care which model produced the numbers.
The same property explains why a purpose-built workflow beat general coding agents in our invoice reconciliation benchmark: model intelligence gets spent only on the parts of the task that are actually ambiguous, which on this workload is reading the page.
Results
What the numbers say
The price of the swap: latency
Open-weight models are served by inference providers whose throughput and queueing behaviour differ from those of the large closed vendors. Across our runs the open-weight configuration takes on the order of twice as long per case as the closed-model configuration for an equivalent score. We are not publishing a precise multiple from this run, because its wall clock was taken on a machine shared with other work and so partly measures the hardware.
What a team gets for that time is data control at roughly level cost. Whether the exchange is worth making comes down to where the workflow sits: an overnight batch absorbs a doubling in per-case time without anyone noticing, while a queue someone is actively working through does not.
How we measured
- Data. 60 cases from a public synthetic set, no customer data. Scanned, image-only proof-of-delivery documents, with their receipt notes and purchase orders. The same set used in our POD benchmark, so the numbers are directly comparable to the ones published there.
- One scorer for every system. The composite weights line-item F1 at 0.70, line-field accuracy at 0.20 and header accuracy at 0.10, computed per case, then averaged across the 60.
- Only the models changed. Same workflow, same deterministic engine, same documents, same scorer, same output contract. The closed-model and coding-agent rows are unchanged from the earlier published run.
- Cost is measured, not estimated. Total LLM spend to complete all 60 cases, counted from token usage at provider list prices, expressed as a multiple of the closed-model configuration's spend.
- Latency is quoted as an approximate figure and not as a benchmark result. This run's wall clock was taken on a shared machine, so we give the direction and rough size of the gap without attaching a precise multiple to a contended measurement.
Limitations
Cost parity is workload-dependent, and this is the most important caveat on the page. Proof-of-delivery processing spends most of its tokens reading pages, with comparatively little multi-step reasoning per case. On a task weighted the other way, we have measured the same swap costing materially more than the closed-model configuration for an equivalent score. The 0.98× belongs to this workload only, and any team considering the switch should measure it on their own document mix.
This is one run of 60 cases. The accuracy figures in Figure 1 are means over 60 cases, one run per system, which is the same basis the earlier published comparison used. Repeat-stability across identical reruns is being measured separately and is not claimed here. In our earlier POD work that measure separated configurations more sharply than accuracy did, so its absence is a real gap in this page.
Provider reliability was visibly worse. The open-weight run lost extractions to repeated bursts of connection failures against the inference provider, and those documents had to be re-processed before the run was complete. Accuracy was unaffected, since a document either gets read or gets retried, but the operational difference between a mature closed API and the current open-weight serving landscape showed up clearly in the run logs.
Deployment posture is a separate question from model choice. That an open-weight model can run on infrastructure you control does not by itself describe how any particular Cadel deployment is configured. Teams solving for data residency should talk to us about the specific arrangement, which this benchmark does not settle.
What comes next
We will keep running this comparison as the open-weight frontier moves, on the same harness, with the same shared-scorer rule. Two gaps are worth closing next. The first is repeat stability across reruns. The second is the latency figure, which we would like to measure on an idle machine, since the current gap looks like a serving constraint and serving constraints tend to ease faster than model quality improves.
