{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1469",
  "slug": "agents-graded-on-real-work",
  "edition": "2026-11-12T12",
  "url": "https://hangingcontext.com/stories/2026-11-12/agents-graded-on-real-work/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "angle": "New benchmarks grade agents on running a store, shopping, playing games and controlling robots, not on chatting.",
  "headline": "Agents are now graded on running a store, shopping and driving robots",
  "dek": "Five preprints published on 11 and 12 November replace chat-style tests with weeks of storekeeping, shopping advice, game play and robot control, and the scores show how far agents still are from finishing the job.",
  "published_at": "2026-11-12T12:43:41Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 5 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 5,
  "publishers": [
    "afterquery-blog",
    "huggingface-daily-papers"
  ],
  "sources": [
    "afterquery-blog",
    "huggingface-daily-papers"
  ],
  "distinct_source_count": 2,
  "key_facts": [
    {
      "text": "In a preprint, StoreBench asks agents to run a simulated online apparel store for 30 to 45 simulated days, using 29 merchant tools.",
      "cites": [
        {
          "claim_ids": [
            "1794458071419584951",
            "1794458071420584952"
          ],
          "manifest_id": "1794457502597583438"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, CARTBench tests shopping advice with 169 timestamped cases across 17 retail domains and 1,328 expert-annotated criteria.",
      "cites": [
        {
          "claim_ids": [
            "1794480283401663763",
            "1794480283405663771"
          ],
          "manifest_id": "1794480045646660363"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, Embodied First Exam tests coding agents on 1,156 tasks across 35 suites and 47 robots, with a top score of 34% on 456 core tasks in its Standard setting.",
      "cites": [
        {
          "claim_ids": [
            "1794424575577434324",
            "1794424575578434326",
            "1794424575580434328",
            "1794424575591434341"
          ],
          "manifest_id": "1794424343619433102"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, GTAGym checks GTA V tasks from engine state, with 103 online tasks and 47 image-based tasks.",
      "cites": [
        {
          "claim_ids": [
            "1794461728587595305",
            "1794461728634595306",
            "1794461728664595307"
          ],
          "manifest_id": "1794461371751594270"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, Interactive Visual Gym reports 315 scenarios and a 53.4% task completion rate for its best model, Gemini 4 Argon.",
      "cites": [
        {
          "claim_ids": [
            "1794480286382664414",
            "1794480286384664416"
          ],
          "manifest_id": "1794480045646660362"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The test has moved to the job itself",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794458071419584951",
                "1794458071420584952"
              ],
              "manifest_id": "1794457502597583438"
            },
            {
              "claim_ids": [
                "1794480283401663763",
                "1794480283398663759"
              ],
              "manifest_id": "1794480045646660363"
            }
          ],
          "markdown": "Five preprints published on 11 and 12 November share one change of method: they grade agents on work that lasts, changes and pushes back, rather than on single answers. In a preprint, StoreBench puts an agent in charge of a simulated online apparel store for 30 to 45 simulated days, with 29 merchant tools to manage the store. In a preprint, CARTBench tests shopping advice on 169 timestamped cases across 17 retail domains, where requests are complex and ambiguous and come with timelines."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794461728498595304",
                "1794461728587595305",
                "1794461728634595306"
              ],
              "manifest_id": "1794461371751594270"
            },
            {
              "claim_ids": [
                "1794424575574434320",
                "1794424575577434324",
                "1794424575578434326"
              ],
              "manifest_id": "1794424343619433102"
            },
            {
              "claim_ids": [
                "1794480286379664411",
                "1794480286382664414"
              ],
              "manifest_id": "1794480045646660362"
            }
          ],
          "markdown": "The same shift runs through the other three preprints. In a preprint, GTAGym is an environment for evaluation and reinforcement learning on GTA V tasks, with 103 online tasks reconstructed across nine task families and 47 image-based shooting and sniping tasks. In a preprint, Embodied First Exam is a framework for coding agents that control robots, with 1,156 tasks, 35 suites and 47 robots. In a preprint, Interactive Visual Gym enables closed-loop evaluation of visual interaction models by jointly simulating task environments and human users, with 315 interaction scenarios across cooking, household manipulation and mobile-interface domains."
        }
      ]
    },
    {
      "heading": "The scores are low where the work is hardest",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794458071409584944",
                "1794458071411584945",
                "1794458071412584946",
                "1794458071413584947",
                "1794458071422584953",
                "1794458071423584954"
              ],
              "manifest_id": "1794457502597583438"
            }
          ],
          "markdown": "In the StoreBench preprint, Claude Fable 5.2 is reported as the leading model, with a mean composite score of 0.835 and an 89% pass rate. That result is the benchmark's own finding, not a product launch, and the preprint also reports a smart-triage baseline at a mean composite score of 0.764 with a 97% pass rate, so the leading model scores higher while passing less often. The preprint names underbuying as a recurring failure: it appears in 178 of 693 runs in the main suite, and incomplete refunds and unresolved returns can lead to escalating fines."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794480283408663781",
                "1794480283410663786",
                "1794480283412663790"
              ],
              "manifest_id": "1794480045646660363"
            },
            {
              "claim_ids": [
                "1794461728675595308",
                "1794461728682595309"
              ],
              "manifest_id": "1794461371751594270"
            },
            {
              "claim_ids": [
                "1794424575590434338",
                "1794424575591434341"
              ],
              "manifest_id": "1794424343619433102"
            },
            {
              "claim_ids": [
                "1794480286384664416"
              ],
              "manifest_id": "1794480045646660362"
            }
          ],
          "markdown": "The other preprints show the same pattern of partial success. In the CARTBench preprint, three of the ten models evaluated exceeded 90% on Core criteria, but none exceeded 45% on Excellence criteria, and the authors describe an understanding-to-execution gap between required behaviour and high-quality purchase advice. In the GTAGym preprint, the best configuration of frontier models reached 70.9% overall accuracy, while aerial navigation, evasion and interception remain difficult. In the Embodied First Exam preprint, GPT-6 Astra in Codex is the strongest of eight agents tested, with a 34% success rate on 456 core tasks in the Standard setting. In the Interactive Visual Gym preprint, Gemini 4 Argon completes only 53.4% of tasks within the time limit."
        }
      ]
    },
    {
      "heading": "How the tests try to stay honest",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794458071407584943",
                "1794458071417584949",
                "1794458071418584950",
                "1794458071415584948"
              ],
              "manifest_id": "1794457502597583438"
            },
            {
              "claim_ids": [
                "1794461728664595307"
              ],
              "manifest_id": "1794461371751594270"
            }
          ],
          "markdown": "Each preprint describes controls meant to stop agents gaming the test. In the StoreBench preprint, agents manage a catalog of 3,448 product variants and source inventory from a market of 40 suppliers, and the benchmark evaluates agents with safeguards against reward hacking. StoreBench uses Medusa v2 as the underlying commerce platform for the simulated store. In the GTAGym preprint, the environment uses automatic engine-state outcome checks to verify task completion, rather than relying on a human to judge whether a game task was finished."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794424575583434332",
                "1794424575585434334",
                "1794424575587434335",
                "1794424575588434337"
              ],
              "manifest_id": "1794424343619433102"
            },
            {
              "claim_ids": [
                "1794480286386664419",
                "1794480286388664420"
              ],
              "manifest_id": "1794480045646660362"
            }
          ],
          "markdown": "The robot and visual tests add settings that change what the agent is allowed to know. In the Embodied First Exam preprint, the Standard setting provides only sensor observations and does not allow environment resets by default, while the Realtime setting keeps the environment running while the agent reasons, and the Privileged setting provides access to the full simulator state and unlimited resets. Coding agents in the framework can choose between direct control, code-to-control, or training a policy to control. In the Interactive Visual Gym preprint, the interaction-quality metric shows 78.5% agreement with human participant preferences, and closed-loop training using GRPO improves task success and interaction quality for the Qwen3.5-4B model in cooking environments."
        }
      ]
    },
    {
      "heading": "What a developer should take from this",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794458071422584953",
                "1794458071423584954"
              ],
              "manifest_id": "1794457502597583438"
            },
            {
              "claim_ids": [
                "1794480283408663781",
                "1794480283410663786"
              ],
              "manifest_id": "1794480045646660363"
            },
            {
              "claim_ids": [
                "1794461728742595313"
              ],
              "manifest_id": "1794461371751594270"
            }
          ],
          "markdown": "Taken together, the five preprints argue for matching the test to the job rather than trusting a single leaderboard. A store test that runs for weeks exposes restocking and refund failures that a short test would miss, and a shopping test that separates Core from Excellence criteria shows that getting the basics right is not the same as giving expert advice. The GTAGym preprint also reports a concrete training result: LoRA reinforcement learning of Qwen3.8-27B on generated on-foot navigation tasks improved performance on native GTA V tasks from 15% to 23%."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794458071411584945"
              ],
              "manifest_id": "1794457502597583438"
            },
            {
              "claim_ids": [
                "1794461728675595308"
              ],
              "manifest_id": "1794461371751594270"
            },
            {
              "claim_ids": [
                "1794480286384664416"
              ],
              "manifest_id": "1794480045646660362"
            },
            {
              "claim_ids": [
                "1794424575591434341"
              ],
              "manifest_id": "1794424343619433102"
            }
          ],
          "markdown": "The limits are part of the finding. None of the five preprints claims agents have mastered these jobs: the best reported rates are 89% passing a store test, 70.9% on a game benchmark, 53.4% completing visual interaction tasks, and 34% on core robot tasks. For a team choosing an agent for real work, the useful question the preprints support is narrower and more practical: which failure modes does the benchmark expose, and do they match the failures that would cost money or time in the team's own setting?"
        }
      ]
    }
  ],
  "unresolved": [
    "Whether StoreBench results transfer to real stores, since the store, suppliers and competitors are simulated.",
    "Whether the Excellence gap in CARTBench reflects model limits or the difficulty of the expert criteria themselves.",
    "How GRPO closed-loop training in Interactive Visual Gym performs outside cooking environments."
  ],
  "decision_card": {
    "not_for": [
      "Choosing a model from a single leaderboard score",
      "Treating a preprint result as a product launch or a vendor claim"
    ],
    "useful_for": [
      "AI developers choosing an agent for store, shopping, game or robot work",
      "Teams picking a benchmark that matches a real job",
      "Researchers comparing failure modes across agent tests"
    ],
    "compare_against": [
      "Vending-Bench and other short-horizon agent tests",
      "Single-turn question answering benchmarks",
      "Vendor demos without a published task set"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-12T12"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1794457502597583438",
    "1794480045646660363",
    "1794424343619433102",
    "1794461371751594270",
    "1794480045646660362"
  ],
  "manifests": [
    {
      "title": "StoreBench: Evaluating AI Agents for Business Operations over Long Horizons",
      "source_url": "https://afterquery.com/blog/storebench",
      "manifest_id": "1794457502597583438",
      "source_name": "afterquery-blog",
      "published_date": "2026-11-11"
    },
    {
      "title": "CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines",
      "source_url": "https://huggingface.co/papers/2611.05862",
      "manifest_id": "1794480045646660363",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-12"
    },
    {
      "title": "Embodied First Exam",
      "source_url": "https://huggingface.co/papers/2611.12177",
      "manifest_id": "1794424343619433102",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-11"
    },
    {
      "title": "GTAGym: Real AAA Game Tasks Environment for Evaluation and Reinforcement Learning",
      "source_url": "https://huggingface.co/papers/2611.13028",
      "manifest_id": "1794461371751594270",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-12"
    },
    {
      "title": "Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds",
      "source_url": "https://huggingface.co/papers/2611.08286",
      "manifest_id": "1794480045646660362",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-12"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1794457502597583438",
      "1794480045646660363",
      "1794424343619433102",
      "1794461371751594270",
      "1794480045646660362"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1794457502597583438",
      "all": "for id in 1794457502597583438 1794480045646660363 1794424343619433102 1794461371751594270 1794480045646660362; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-12/agents-graded-on-real-work/",
    "json": "https://hangingcontext.com/stories/2026-11-12/agents-graded-on-real-work/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-12/agents-graded-on-real-work/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-12T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-15/ai-prs-broke-open-source-review/",
      "headline": "AI writes the code now. Humans can't read it fast enough — and open source just proved it",
      "slug": "ai-prs-broke-open-source-review"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-15/agents-claim-work-they-never-did/",
      "headline": "Agents claim work they never did. Now we can count how often",
      "slug": "agents-claim-work-they-never-did"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-14/inference-throughput-race-kimi-deepseek/",
      "headline": "Throughput is the new model spec, and the gap between hosts is now fortyfold",
      "slug": "inference-throughput-race-kimi-deepseek"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-14/coding-agent-orchestration-three-patterns/",
      "headline": "Coding agents split three ways, and the choice is now about control, not capability",
      "slug": "coding-agent-orchestration-three-patterns"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Run the cited Manifests yourself",
    "text": "Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.",
    "cta": {
      "label": "Get a key",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-12/agents-graded-on-real-work/"
  }
}