---
title: Before Agents Ship, Teams Are Learning to Measure the Work Itself
description: Qodo, Patronus, Mastra and Google all shipped measurement tooling this week, and the common thread is a shift from judging models to judging the work agents actually do against specifications, training tasks and production traces.
canonical: https://hangingcontext.com/stories/2026-10-29/agent-work-gets-measured-first/
last-updated: 2026-10-29
url: https://hangingcontext.com/stories/2026-10-29/agent-work-gets-measured-first/
published: 2026-10-29T12:38:43Z
edition: 2026-10-29T12
newsroom: artificial-intelligence
beat: developers
trust: confirmed
source_count: 4
cited_manifest_ids: ['1793271971130664358', '1793246009239100759', '1793210870661097325', '1793226597395489566']
---

# Before Agents Ship, Teams Are Learning to Measure the Work Itself

*Qodo, Patronus, Mastra and Google all shipped measurement tooling this week, and the common thread is a shift from judging models to judging the work agents actually do against specifications, training tasks and production traces.*

Confirmed · 4 sources · Artificial Intelligence · Published 2026-10-29 12:38 UTC

## Key facts

- Qodo can now follow links from pull requests through tickets to specifications and flag code that passes tests yet misses requirements. [cites: 1793271971130664358]
- Patronus reports reinforcement learning alone nearly matched supervised fine-tuning plus RL on GitHub and Unreal tasks. (self-reported) [cites: 1793246009239100759]
- Mastra span queries filter and group individual spans across traces, with up to 100 conditions per query. [cites: 1793210870661097325]
- Google Agents CLI supports continuous testing, prompt regression detection and threshold-based alerting for agent quality. [cites: 1793226597395489566]

## Does the code match the spec

Qodo can now follow links from pull requests to tickets and then to specifications, reviewing code against those requirements, with spec sources including Asana, Confluence, Notion, ServiceNow and JSM Assets. The problem it targets is specific: pull requests can pass compilation and tests while still missing critical requirements defined in linked specifications, and the company says its review catches code that functions correctly but fails to deliver the intended behavior or constraints. [cites: 1793271971130664358]

The security version of that gap is concrete: Qodo flags missing account-level authorization checks as security issues when the specification requires them but the code omits them. The capability also integrates into the Kiro spec-driven workflow, where specifications become plans, tasks and code, so the check sits inside the development loop rather than arriving as a separate audit after the fact. [cites: 1793271971130664358]

## What is each training task worth

Patronus AI built an internal post-training stack to quantify the value of training tasks, automating five stages from training and checkpointing through endpoint deployment, evaluation and leaderboard updates. In experiments using Qwen3.8-27B as the base model, supervised fine-tuning followed by reinforcement learning lifted GitHub task performance from 35.6 percent to 47.8 percent, while reinforcement learning alone reached 45.4 percent. [cites: 1793246009239100759]

> **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start)


The Unreal Engine results point the same way: fine-tuning plus reinforcement learning moved performance from 64.3 percent to 71.5 percent, while reinforcement learning alone reached 73.1 percent, and the team says reinforcement learning alone performed about as well as the combined approach. The team audited 50 passing runs to verify no reward hacking occurred, and plans a data flywheel to store and analyze trajectories, which suggests the stack is meant to price training effort rather than to celebrate any single run. [cites: 1793246009239100759]

## Watching work in production

Mastra now supports span queries that filter and group individual spans across traces, which are more granular than trace queries because they search within each trace and return matching spans. Queries can combine up to 100 conditions, filter by time range, type, name, status, error and duration, return up to 1,000 spans per page, and aggregate matching spans into measures such as counts, error rates and latency. [cites: 1793210870661097325]

Google Agents CLI evaluation loop covers the same need from the testing side, with continuous testing and grading designed to catch prompt regressions, side-by-side trace comparisons for model backend swaps, nightly scheduled runs and threshold-based alerting to track quality over time. Taken together, the four releases describe a pipeline: check the work against the spec, price the training task, then watch the traces and alert when quality drifts, instead of discovering drift from users. [cites: 1793226597395489566]

## The decision this creates

The practical decision these tools force is a budget question disguised as an engineering question: which agent work deserves more training, more review, or retirement. Patronus plans a data flywheel to store and analyze trajectories and is exploring partial credit rewards and adaptive task choice to optimize compute, while Google threshold alerting tracks quality over time so drift triggers action rather than anecdote. A team that can price a training task, check code against its spec, and watch production spans has three independent signals before it scales any agent further. [cites: 1793246009239100759, 1793226597395489566]

Reproducibility is the quieter half of the measurement story: Patronus snapshots code and records git commits for reproducibility and links runs to Weights and Biases pages, while Mastra span queries work on self-hosted projects using DuckDB, PostgreSQL or ClickHouse. Measurement that cannot be rerun or replayed from the same traces will not survive an audit, which is why both teams put the record-keeping next to the metric. [cites: 1793246009239100759, 1793210870661097325]

## What is still unresolved

- Whether spec-linked review keeps working once specifications themselves go stale.
- How far reinforcement-learning-only training generalizes beyond the GitHub and Unreal tasks tested.

## Decision card

- **Useful for:** Teams scaling agents past demos into production workflows
- **Not for:** Teams evaluating a model on a public benchmark alone
- **Compare against:** Manual code review; Leaderboard-only model selection

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1793271971130664358` | How Qodo Uses Linked Specs to Catch Missing Requirements | [qodo-publishing](https://qodo.ai/blog/how-qodo-uses-linked-specs-to-catch-missing-requirements) | 1793272282073507485, 1793272282071897269, 1793272282079297379, 1793272282080670405, 1793272282077743250, 1793272282075790557, 1793272282076021588 | 10-29 00:00 |
| `1793246009239100759` | We Built Our Own Post-Training Stack to Measure What Our Tasks Are Worth | [patronus-ai-blog](https://patronus.ai/blog/post-training-stack) | 1793246230695920062, 1793246230696983013, 1793246230700164292, 1793246230689786189, 1793246230702821118, 1793246230707090718, 1793246230693736758, 1793246230698546833, 1793246230710035115, 1793246230715153596, 1793246230716843070, 1793246230718703020, 1793246230712335200 | 10-29 00:00 |
| `1793210870661097325` | Introducing Span Queries | [mastra-publishing](https://mastra.ai/blog/introducing-span-queries) | 1793211043689423856, 1793211043696095769, 1793211043693731657, 1793211043691097148, 1793211043701211103, 1793211043695483736 | 10-28 00:00 |
| `1793226597395489566` | Hill-Climbing Agent Quality with Agents CLI Evaluation | [google-developers-cloud](https://developers.googleblog.com/hill-climbing-agent-quality-with-agents-cli-evaluation) | 1793226987534141109, 1793226987531075814, 1793226987555972032, 1793226987523452629, 1793226987539985158, 1793226987547818046 | 10-28 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1793271971130664358`
- `1793246009239100759`
- `1793210870661097325`
- `1793226597395489566`

```sh
for id in 1793271971130664358 1793246009239100759 1793210870661097325 1793226597395489566; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Run the cited Manifests yourself. Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. [Get a key](https://synorb.com/connect/start)

## Related stories

- [Agent Benchmarks Are Starting to Grade the Work, Not Just the Final Answer](https://hangingcontext.com/stories/2026-10-30/agent-benchmarks-check-the-work-not-just-outcome/)
- [Inference Is Getting Cheaper From Scheduling, Not From Bigger Cards](https://hangingcontext.com/stories/2026-10-29/inference-wins-move-to-scheduling/)
- [Clinical AI Is Being Graded on Guidelines, and the Grades Do Not Travel Well](https://hangingcontext.com/stories/2026-10-29/clinical-llm-benchmarks-miss-guidelines/)
- [Thirty Agents, Eighteen Pipelines: Reproducibility Is Now the AI Problem](https://hangingcontext.com/stories/2026-10-29/agentic-science-needs-an-audit-trail/)

## Links

- HTML: https://hangingcontext.com/stories/2026-10-29/agent-work-gets-measured-first/
- JSON: https://hangingcontext.com/stories/2026-10-29/agent-work-gets-measured-first/index.json
- Markdown: https://hangingcontext.com/stories/2026-10-29/agent-work-gets-measured-first/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-10-29/agent-work-gets-measured-first/. Terms: https://hangingcontext.com/terms/reuse/