Debugging agents means querying traces, not scrolling logs

Sampling breaks down

Traditional tracing products sample heavily and discard traces quickly, Braintrust says, and that approach was designed for deterministic, multi-service software, where the goal was understanding bottlenecks. Agent work is different: agent traces serve as the authoritative record for how an agent behaves during long-running, probabilistic tasks. When an agent's answer depends on a chain of tool calls and intermediate steps, a sampled log leaves the team guessing about the steps that were thrown away. m:…0390

Braintrust's answer is a Trace Lake that supports fast ingest, SQL-like querying of nested fields and continuous analysis, the company says, with Topics to classify traces, Patterns to investigate recurring issues and Loop automations for scheduled investigations. Its Nitro engine separates waiting for storage from processing the data it returns; the company says full-text searches in production A/B tests ran more than twice as fast as previous iterations, and other queries 15% faster. Notion's AI modeling team uses Brainstore to run searches in seconds that previously took hours, the company says. m:…0390

Humans still have to read the thread

Querying only helps if someone can judge what the trace shows. LangChain argues human expertise is required to identify quality and explain why an interaction succeeded or failed, and reviewing full threads lets experts see user intent and the agent's decision path. An agent trace records user messages, model responses, tool calls and intermediate steps. Flagging a trace provides a signal but does not explain how to change agent behavior, a distinction that matters when a team must decide what to fix. m:…6556

LangSmith lets queue items be single runs or full multi-turn threads, and human-labeled examples can align LLM judges, update online evaluators and build golden datasets for continuous testing. Fetch Rewards uses annotation queues to assess its AI Rewards Assistant, the company says, with reviewers inspecting trace metadata to verify tool usage and data grounding, checking product-card images against represented products and confirming offer availability for specific users. Engineers then use reviewer reasoning to tune automated judge logic. m:…6556

Traces that other systems can query

Databricks' MLflow now takes native OpenTelemetry export from the Claude Platform, the company says: Claude's instrumentation generates spans exported using the OpenTelemetry Protocol, Databricks receives them through Zerobus Ingest and stores them in Unity Catalog tables linked to an MLflow experiment, where Unity Catalog controls access and the data can be queried with SQL. MLflow's Analyze trace action uses Genie Code to investigate agent executions, Issues in beta uses it to identify recurring quality and operational problems, and Review Queues let domain experts assess recorded requests and responses. m:…1644

Andesite pushes the same idea further into security work: its agents form hypotheses, collect supporting or contradicting claims and link every claim to specific query results, with an evidence panel in the chat interface and audit logs recording high-level system and user actions, the company says. It cites a figure that 81% of CISOs worry AI agents have too much access. This suggests debugging is becoming less about reading logs and more about querying evidence, but the evidence is only as good as the traces teams keep, and the tools for keeping them are still largely sold as vendor platforms. m:…7612