arXiv:2608.02441 2026

Agentic Commerce World

An Auditable and Verifiable Environment for Vibe Commerce

Shicheng FanMingdai YangDuohao WangCanyu Chen Yongfeng ZhangHua WeiManling LiJulian McAuley Kun ZhangPhilip S. YuKejing YuZhiwei Liu

UIC · Springbrand · Northwestern · Rutgers · ASU · UC San Diego · CMU · MBZUAI · Microsoft AI

Paper arXiv Code Benchmark
ACWorld architecture connecting buyer and merchant agents, VCP tool servers, the commerce platform, shared state, and verifiable evaluation.
ACWorld separates agent control, protocol validation, platform execution, and evaluation while preserving an end-to-end evidence chain.

Overview

Abstract

In vibe coding, human intent is executed by agents through tools; we envision an analogous vibe commerce paradigm in which independently controlled Buyer and Merchant agents transact on behalf of humans and organizations.

We introduce Agentic Commerce World (ACWorld), an auditable and verifiable environment where a Vibe Commerce Protocol (VCP) validates operations before updating shared state and records every interaction. This design makes commerce workflows reproducible, attributable, and open to process-level evaluation—not just final-answer scoring.

The benchmark contains a 200-task capability track and a 60-task large-catalog track over 785,022 transactable listings. Across ten models, mean scores range from 65.9–85.6 on capability tasks and 56.1–91.4 on large-catalog tasks, revealing that strong outcomes do not always imply sound process evidence.

Why it matters

What ACWorld contributes

01

Protocol attribution

Every validated action is tied to the responsible buyer or merchant agent, creating a replayable audit trail.

02

Persistent shared market

Agents interact through a common world state with inventory, payments, orders, fulfillment, returns, and disputes.

03

Evidence-linked evaluation

Outcomes are scored together with process evidence, exposing hidden shortcuts and unsupported success.

Evaluation suite

Two tracks, one auditable benchmark

The suite covers focused protocol capabilities and realistic search at scale, with tasks spanning both sides of the marketplace.

Ring chart showing the composition of the 260-task ACWorld benchmark across the capability and large-catalog tracks.
Capability track 200 tasks

Targeted coverage of protocol behaviors across ten task families.

Capabilities
80
Buyer / Merchant
116 / 84
World snapshots
1,082
Large-catalog track 60 tasks

Realistic shopping and merchant workflows over a marketplace-scale catalog.

Capabilities
18
Buyer / Merchant
45 / 15
Listings
785,022

Experiments

Outcome quality varies—and evidence matters

Ten frontier and open models were evaluated. Aggregate scores show strong progress, while outcome attribution exposes where seemingly successful runs lack the required procedural support.

Stacked bar chart comparing full-credit, partial-credit, and zero-credit outcomes for evaluated models.
Outcome distribution across evaluated models and benchmark tracks.

Key finding

Final state alone is not enough.

Of 861 non-full-credit capability runs, 99 reached a final state also produced by a full-credit execution of the same task. ACWorld’s evidence trail distinguishes genuine completion from unsupported or procedurally flawed success.

85.6%Best capability score
91.36%Best catalog score
97.8%5×5 market optimum
Heatmap showing benchmark deficits by task family across evaluated models.
Task-family deficits reveal complementary failure patterns across models.

Try the benchmark

Quick start

Full documentation ↗
Shell 
git clone https://github.com/shichengf/ACWorld.git
cd ACWorld

./run_benchmark.sh run \
  --tasks 200 \
  --model google/gemini-3.6-flash \
  --api-key-file /absolute/path/to/openrouter-key.txt

Reference

Citation

View on arXiv ↗
BibTeX 
@misc{fan2026agenticcommerceworld,
  title         = {Agentic Commerce World: An Auditable and
                   Verifiable Environment for Vibe Commerce},
  author        = {Fan, Shicheng and Yang, Mingdai and Wang, Duohao and
                   Chen, Canyu and Zhang, Yongfeng and Wei, Hua and
                   Li, Manling and McAuley, Julian and Zhang, Kun and
                   Yu, Philip S. and Yu, Kejing and Liu, Zhiwei},
  year          = {2026},
  eprint        = {2608.02441},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2608.02441}
}