◇ The problem: an agent needs the price the shop will actually charge
An AI agent that buys for someone has to answer one question before anything else: what will this cost, delivered to this address? Product price, tax, shipping for that postcode, minimum order rules, mandatory fees. Catalog prices alone cannot prove a delivered total, and a total that turns out wrong is a broken promise to the buyer.
KaliCart’s job is narrow: represent the shop’s truth to agents as faithfully as the shop itself would, and get there faster, with fewer requests. Whether the buyer likes that truth is not our business. This benchmark tests whether two scripted agents can return an offer with a WooCommerce-calculated total, and at what cost in time and requests.
◎ The setup: two agents, the same tasks, one independent referee
Both agents are deterministic scripts, not language models. Each task is a fixed shopping scenario generated from catalog products: a search word, an Italian delivery address, a budget for the delivered total and, where it applies, a size or colour. “Find X, delivered to postcode P, at most B euros all in. Answer with an available offer, its proven total and a checkout path, or abstain.”
| Agent A — direct | Agent B — KaliCart | |
|---|---|---|
| How it searches | Calls each shop’s WooCommerce Store API, shop by shop | One search across the KaliCart network |
| How it gets the total | Opens a real cart on the shop and reads the total | Asks the shop’s Bridge for a quote, calculated by that shop’s WooCommerce |
| What it is told | The URLs of the nine shops, with full Store API access | No shop URLs (KaliCart already indexes the participating shops) |
| When it answers | Only with a total from a cart | Only with a total WooCommerce calculated (estimates rank candidates, never answer) |
A third component, the referee, checks every answer independently: it opens a fresh WooCommerce cart for the proposed product and address and compares totals to the cent. An answer counts only if the product matches the need, is in stock, and the total matches the shop’s own cart within the budget. A wrong total is a false positive, the worst outcome.
The shops are nine Italian WooCommerce stores in different sectors: eight independent merchants and our own test store. They are not named here. Both agents ran in alternating order, with at least 2 seconds between requests to the same shop. No order, payment or customer account was ever created.
≡ The rules, written down before the run
Every threshold was committed to the repository before this round ran, and none was changed after seeing results. The verdict is taken on a set of 18 tasks generated with a fresh seed, from products never used in earlier rounds.
| Question | Rule for KaliCart (B) against direct (A) |
|---|---|
| ≈ Tie on correctness | B closes at least as many tasks as A minus one, with zero false positives |
| ◷ Win on time | Median net time per task at least 50% lower |
| ↓ Win on shop load | Requests to shops at least 80% lower |
| ↺ Win on calls | Median calls per task no higher than A |
| ▢ Win on carts | Carts at most 10% of A’s |
The idea behind them: tie where only a tie is possible, win where there is margin. Nobody expects KaliCart to be more correct than the shop; it must be exactly as correct, and better on the way there.
▤ Results on 18 never-seen tasks
KaliCart closed 2 more tasks than the direct agent, gave no wrong total in this sample, and took about a fifth of the time.
| Measure | A — direct | B — KaliCart | Rule | Verdict |
|---|---|---|---|---|
| Tasks closed | 13 / 18 | 15 / 18 | B ≥ A − 1, 0 false positives | ✓ |
| False positives | 0 | 0 | 0 | ✓ |
| Median net time per task | 9.3 s | 1.8 s (81% less) | at least 50% less | ✓ |
| Median calls per task | 12 | 7 | no more than A | ✓ |
| Requests to shops | 219 | 44 (79.9% less) | at least 80% less | ✗ by one request |
| Carts | 19 | 22 | at most 10% of A | ✗ (see below) |
Two things stand out in the detail:
- Where the direct agent stops, KaliCart goes on. In three tasks the shop’s own Store API search returned no candidate for the need, while the network search found the product. In two tasks of the repeated sets, every cart the direct agent opened was refused by WooCommerce (cart validation rules), and KaliCart found an orderable offer.
- The only task the direct agent closed and KaliCart did not on the never-seen set was one where no WooCommerce quote fell within the budget; KaliCart abstained rather than guess.
On the two sets already seen in earlier rounds the picture is the same: 18 / 18 against 16, and 16 against 13, with 82% fewer requests and 82–83% less time.
↗ How we got there: from 8 to 18 out of 18
On the same 18 tasks, KaliCart went from 8 closed to 18 in four rounds over three days. Each step came from finding why a task failed and fixing the cause for every shop, never a patch for one shop.
| Round | KaliCart closed (same 18 tasks) | Cause found and fixed |
|---|---|---|
| 4 — 29 Oct | 18 / 18 | The quote now runs WooCommerce’s own cart checks (cart validation rules) and keeps shop-added cart lines; the network search filters prices by currency and says which shops offer a quote |
| 3 — 29 Oct | 16 / 18 | The quote’s temporary cart now offers the full WooCommerce session interface, so pricing and discount plugins that read the session work |
| 2 — 28 Oct | 14 / 18 | Quotes and checkout handoff on by default; the quote trusts WooCommerce’s shipping calculation instead of pre-checking shipping methods; a shipping zone with methods but no locations no longer reads as “not delivered” |
| 1 — 27 Oct | 8 / 18 | First strict run (only WooCommerce-calculated totals count): most rejections came from over-strict pre-checks on shipping methods and from how shipping zones were read; one shop’s quote was unavailable |
The direct agent stayed between 16 and 17 on these tasks throughout. What changed was KaliCart’s fidelity to what each shop’s WooCommerce actually does.
△ What did not pass, and a mistake in our own design
Two rules were not met, and we report them as written.
Requests to shops: 79.9% fewer, against a rule of 80%. 44 requests against 219 is one request above the line (the limit was 43.8). On the two repeated sets the rule was met at 82%. In practice requests to the shops fell by four fifths; by the letter of the rule, it is a no.
Carts: a rule that could not be met, and we wrote it. The rule counted every “cart” the same way, but the two agents do very different things:
| Direct agent’s cart | KaliCart quote | |
|---|---|---|
| What it is | A real WooCommerce customer session | A cart built in memory inside one request |
| Persistence | A Store API customer session, persisted according to the shop’s configuration | A temporary in-memory cart; its WordPress database writes are blocked |
| Customer session | Yes | Not created |
| Requests to build it | Several (create, address, add item, shipping) | One |
| Afterwards | A customer session the shop keeps until it expires | No customer session |
Every KaliCart answer rests on a quote, so the rule asked KaliCart to answer without letting WooCommerce calculate the total, which is the one thing it must never do. The rule was wrong by design. Future rounds will count persisted customer sessions and database writes instead, fixed before measuring.
○ Limits, and what comes next
This is a first measured lead, not a general law. What it does not show:
- Scale. Nine shops, 18 tasks per set, one run per set. “No wrong total” means 0 observed in 54 tasks, not a guarantee.
- The baseline. Agent A is a scripted agent using the shops’ public Store API, not every way an agent could use WooCommerce.
- Cost to the shop. A separate measurement of CPU and database work per request was inconclusive because of two flaws in our instrument. We do not claim lower server cost yet.
- Familiar shops. The fixes were found on these nine shops. The test that matters next is the same benchmark, with the same rules, on shops and tasks we have never seen.
That is the plan: when new merchants join the network, the benchmark runs again on them, rules fixed in advance, results published as they come, including the no’s.