--- title: The new tests grade agents on real work — and the scores are humbling description: FrontierCUA's best model passes 25% of enterprise workflows. OpenFrontiers finds closed-book rediscovery is rare. And a math modeling bench checks answers against Lean 4 proofs. canonical: https://hangingcontext.com/stories/2026-11-07/new-tests-grade-real-work/ last-updated: 2026-11-07 url: https://hangingcontext.com/stories/2026-11-07/new-tests-grade-real-work/ published: 2026-11-07T12:41:09Z edition: 2026-11-07T12 newsroom: artificial-intelligence beat: research trust: confirmed source_count: 5 cited_manifest_ids: ['1794009412879511031', '1793997239974447412', '1793997239973447409', '1794024750852599646', '1793997239974447410'] --- # The new tests grade agents on real work — and the scores are humbling *FrontierCUA's best model passes 25% of enterprise workflows. OpenFrontiers finds closed-book rediscovery is rare. And a math modeling bench checks answers against Lean 4 proofs.* Confirmed · 5 Manifests · Artificial Intelligence · Published 2026-11-07 12:41 UTC ## Key facts - FrontierCUA's best model, GPT-6.1 Sol, achieves a 25% pass rate on 40 enterprise workflows, at about $6 per rollout, and 23 of the 40 tasks are solved by no model. [cites: 1794009412879511031] - In the OpenFrontiers benchmark, closed-book rediscovery of solutions is rare, occurring in at most 7 of 40 resolved problems, and live-web success mostly reflects retrieval of already published solutions. [cites: 1793997239974447412] - TB-MathModel uses 18 real contest problems and 624 kernel-checked Lean 4 obligations; GPT-6 Astra scored 64.58% while a much cheaper open model reached 59.81%. [cites: 1793997239973447409] - In a preprint, a latent diffusion world model called ADWM estimates LLM agent performance from pre-collected data, without running the agent live in the environment. (self-reported) [cites: 1794024750852599646] ## Enterprise work Collinear AI's FrontierCUA is a benchmark for computer-use agents in enterprise environments, and it is deliberately hard. It tests agents on 40 multi-app workflows across 11 enterprise domains, using high-fidelity clones of commercial software such as Zendesk and HubSpot, with every workflow spanning at least two applications and 16 of the 40 requiring interaction with four or five apps. The best-performing model, GPT-6.1 Sol, achieves a 25% pass rate at a cost of approximately $6 per rollout, and 23 of the 40 tasks are solved by no model at all. The median task runs 374 steps over 36 minutes. [cites: 1794009412879511031] The failure modes are the useful part. For GPT-6.1 Sol, inadequate evidence gathering accounts for 39% of lost rubric points, reasoning errors for 30%, and stopping before finishing for 24%. Interface errors are 5% and arithmetic slips 2%. The benchmark's message is that the bottleneck is not clicking buttons. It is gathering the evidence, reasoning over it, and finishing the job. That is the state of the art for enterprise computer use, priced per attempt: the leading model passes a quarter of the workflows at about $6 a rollout. [cites: 1794009412879511031] ## Science, graded honestly OpenFrontiers, a new cross-disciplinary resource for evaluating LLM agents on research-level problem solving, combines a corpus of 12,151 open research problems with a benchmark of 200 recently resolved mathematics problems, of which 110 were resolved in 2026 and 90 are still open. Seven LLM agents were evaluated. The headline finding is humbling: closed-book rediscovery of solutions is rare, occurring in at most 7 of 40 resolved problems, and live-web success in the benchmark mostly reflects the retrieval of already published solutions rather than genuine problem-solving. Each problem has a Lean 4 statement checked by automated probes and two independent reviews, and grading uses two independent checkers in an isolated sandbox. [cites: 1793997239974447412] > **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start) A second benchmark, TB-MathModel, makes the same point for mathematical modeling. It uses 18 real-world contest problems from 2023 to 2025, with a two-layer feasibility verification system including 624 kernel-checked Lean 4 obligations. GPT-6 Astra achieved a 64.58% accuracy score, Fable 5.1 reached 63.1%, and a significantly cheaper open model reached 59.81% at one hundredth the cost of top models. As the benchmark's authors put it: "Reporting a number is not enough; it has to survive the physics." The gap between the best model and the cheapest one is small, which is itself a finding: on verifiable work, the frontier premium is thin. [cites: 1793997239973447409] ## Grading without running A preprint this week offers a way to grade agents without the cost and risk of running them live. ADWM, an autoregressive diffusion world model, enables off-policy evaluation of LLM agents using pre-collected data. It models each transition as an independent denoising process, which allows step-by-step rollouts, and the agent under evaluation directly guides the diffusion generation at each step through a policy-conditioned score function. The authors report accurate value estimates and evaluation reliability across diverse multi-turn agent tasks. A related framework, SwarmEvolve, uses swarms of hundreds of small latent-reasoning agents that share information by passing KV caches rather than decoding to text, reaching best scores in a median of 3 to 8 minutes against 60 minutes for other state-of-the-art frameworks. [cites: 1794024750852599646, 1793997239974447410] The pattern across the four pieces is a maturing evaluation culture. The tests are getting closer to real work: multi-app enterprise workflows, open research problems, contest modeling with machine-checked proofs. And the scores are getting more honest: 25% on enterprise workflows, rare closed-book rediscovery, a thin gap between the most expensive model and the cheapest one. The yardstick is not failing. It is finally measuring the right thing. [cites: 1794009412879511031, 1793997239974447412, 1793997239973447409] ## What is still unresolved - FrontierCUA's future performance on expanded domains such as MLOps and data engineering is unresolved. - No code or dataset release was explicitly mentioned for ADWM in the material reviewed. - Whether SwarmEvolve's latent-swarm approach generalizes beyond the mathematical discovery tasks tested is unresolved. ## Decision card - **Useful for:** teams choosing agents for enterprise workflows; researchers measuring agent capability honestly - **Not for:** anyone who needs a benchmark score as a buying decision on its own - **Compare against:** your own task success rates on real workflows; prior benchmark generations for the same models ## Provenance ledger | Manifest id | Title | Source | Claims used | Published | |---|---|---|---|---| | `1794009412879511031` | Introducing FrontierCUA: Hard but Fair Benchmark for Evaluating Enterprise Computer Use | [collinear-ai-research](https://blog.collinear.ai/p/frontiercua) | 1794009697858514213, 1794009697860514214, 1794009697864514217, 1794009697853514210, 1794009697855514211, 1794009697857514212, 1794009697863514216, 1794009697865514218, 1794009697866514219, 1794009697872514223, 1794009697868514220, 1794009697870514222 | 11-06 00:00 | | `1793997239974447412` | OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery | [huggingface-daily-papers](https://huggingface.co/papers/2610.37739) | 1793997535057450702, 1793997535058450703, 1793997535047450695, 1793997535049450696, 1793997535050450697, 1793997535052450698, 1793997535053450699, 1793997535055450701 | 11-06 00:00 | | `1793997239973447409` | Terminal-style Mathematical Modeling Bench: The First Agentic and Verifiable Benchmark for Mathematical Modeling | [huggingface-daily-papers](https://huggingface.co/papers/2611.08237) | 1793997535473450795, 1793997535475450796, 1793997535477450797, 1793997535480450799, 1793997535478450798 | 11-06 00:00 | | `1794024750852599646` | Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents | [arxiv-frontier-methods-select](https://arxiv.org/pdf/2606.05558v3) | 1794025319122601657, 1794025319138601661, 1794025319133601659 | 11-06 00:00 | | `1793997239974447410` | SwarmEvolve: Latent Agent Swarms for Evolutionary Discovery and Optimization | [huggingface-daily-papers](https://huggingface.co/papers/2611.08483) | 1793997535190450738, 1793997535191450739, 1793997535196450743, 1793997535198450746 | 11-06 00:00 | ## Run these Manifests yourself Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them. 1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. 2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer `. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief. 3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them. Cited Manifest ids: - `1794009412879511031` - `1793997239974447412` - `1793997239973447409` - `1794024750852599646` - `1793997239974447410` ```sh for id in 1794009412879511031 1793997239974447412 1793997239973447409 1794024750852599646 1793997239974447410; do curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id" done ``` > **From Synorb** — Run the cited Manifests yourself. Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. [Get a key](https://synorb.com/connect/start) ## Related stories - [Lab agents can run the experiment. Novices still cannot follow them.](https://hangingcontext.com/stories/2026-11-07/lab-science/) - [Agents are learning to improve themselves, in swarms and in training](https://hangingcontext.com/stories/2026-11-07/agents-that-improve-themselves/) - [Nobody wants one model anymore: routers and multi-model habits are the hedge against lock-in](https://hangingcontext.com/stories/2026-11-07/model-lock-in-hedge/) - [Diffusion language models get their own training stack — RL, correction, and faster decoding](https://hangingcontext.com/stories/2026-11-07/diffusion-language-models-grow-up/) ## Links - HTML: https://hangingcontext.com/stories/2026-11-07/new-tests-grade-real-work/ - JSON: https://hangingcontext.com/stories/2026-11-07/new-tests-grade-real-work/index.json - Markdown: https://hangingcontext.com/stories/2026-11-07/new-tests-grade-real-work/index.md Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify. --- Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-07/new-tests-grade-real-work/. Terms: https://hangingcontext.com/terms/reuse/