AI at work, no pilot needed: measured evals, travel agents, a voice PoC, and bots closing tickets
Snowflake tells teams to measure agent behavior, not answers; Spotnana ships a multi-agent travel desk; Ashby governs AI interview notes; Toyota Finance pilots voice AI; and x.ai's bots close incident tickets in batches.
- Snowflake's Cortex platform added native agent evaluation and observability; its guidance is to evaluate the full trajectory — tool selection, planning, efficiency — not just final answers, with tool-selection accuracy the top metric. Self-reported m:…4354
- The company recommends golden datasets of 20–50 examples, ideally from production logs in AI_OBSERVABILITY_EVENTS, with serverless async eval runs driven by a YAML configuration file in a Snowflake stage. Self-reported m:…4354
- Spotnana launched a multi-agent AI architecture for travel agency workflows: agents mapped to specific Spotnana APIs under a centralized orchestration layer, with an open design for customer- and partner-built agents. Self-reported m:…9841
- Ashby added two AI Notetaker settings: an org-wide toggle for AI auto-fill in feedback forms, and a switch controlling interviewers' personal instructions and prompts. Self-reported m:…2350
- Toyota Finance begins a November 2026 proof of concept of IVRy's conversational AI platform at its merchant desk — its first generative-AI phone touchpoint — testing AI first response and voice-driven Q&A generation. Self-reported m:…5770
- x.ai's Grok Bot teammates IncidentWatch and IncidentSweep monitor ticket spikes and batch-close incident tickets; on September 18, IncidentSweep cleared 204 tickets in about 10 minutes with a support engineer reviewing and approving the batch, per the company. Self-reported m:…9314
Measure behavior, not answers
Snowflake's Cortex platform now ships native evaluation and observability for AI agents, and its guidance is blunt: deploying Cortex Agents without an evaluation framework means burning credits on unmeasured outputs. The framework's core idea is that evaluating an agent means evaluating the whole trajectory — the right tool, the correct call, an efficient path — not just the final answer. m:…4354
The recommended practice starts small: a golden dataset of 20 to 50 examples, enough to reveal failure patterns without becoming a project of its own, ideally sourced from real production logs captured in AI_OBSERVABILITY_EVENTS. Tool-selection accuracy is the highest-priority metric, the company says, while retrieval metrics like groundedness apply only to the Cortex Search tool step — sub-checks, not headline agent metrics. m:…4354
Evaluation runs execute serverless and asynchronously, driven by a YAML configuration file in a Snowflake stage, with results carrying input, output, ground truth, metric score, evaluation type, duration, and an explanation per score. The machinery exists; the message is that nobody should be flying blind. m:…4354
The agent travel desk
Spotnana launched a multi-agent AI architecture for travel agency workflows: agents mapped to specific Spotnana APIs, working under a centralized orchestration layer, with an open design so customers and partners can develop their own agents and integrate with other systems. m:…9841
The first agents cover the unglamorous core of agency work: automating routine servicing when airline schedule changes cancel segments, validating unticketed segments and issuing change orders, ingesting externally booked trips into the platform, and a co-pilot that suggests responses and summarizes conversations. A fourth agent produces analytics on servicing tasks, including how many the AI servicing agent completed. More capabilities for travelers and travel managers are promised over the coming months. m:…9841
The interview loop gets guardrails
Ashby added two settings to its AI Notetaker: an org-wide toggle controlling AI auto-fill in feedback forms, and a switch for whether interviewers can set personal instructions and prompts. Toggling the personal-instructions switch off removes interviewers' access to their personal auto-fill instructions page; toggling the org switch off disables auto-fill across all interview feedback. m:…2350
It is a small feature with a large signal: AI in hiring is moving from experiment to governed infrastructure, with administrators — not individual interviewers — setting the defaults. m:…2350
A voice PoC at Toyota Finance
Toyota Finance will begin a proof of concept in November 2026 using IVRy's conversational AI platform at its merchant desk — the first time the company is using generative AI at a phone-based customer touchpoint. The PoC tests AI first-response handling and voice-recognition Q&A, where call data is analyzed to generate and recommend new Q&A entries. m:…5770
The target is the familiar pain of push-button IVR: the effort of navigating menus, wait times, and the difficulty of reaching the right window among several. The PoC will verify AI quality and effectiveness, acceptance of AI handling, and whether voice-data analysis feeds continuous improvement. The platform operates around the clock; Toyota Finance frames the effort within its company-wide digital and AI transformation, and cites the Toyota Way ideas of seeing things firsthand and continuous improvement. m:…5770
Bots that own the incident queue
At x.ai, two Grok Bot teammates now own parts of the incident-response job. IncidentWatch reviews the previous 15 minutes of support tickets every five minutes, and when it spots a spike around the same problem it alerts the on-call engineer with the apparent issue, the number of matching tickets, and examples. IncidentSweep handles the other end: identifying affected tickets and closing them in an approved batch. m:…9314
The human stays in the loop where it counts: before sending anything, IncidentSweep shows the engineer the proposed batch, the included tickets, and the exact message customers will receive, then sends approved messages in small batches and records each action — stopping and reporting if anything fails. During one incident on September 18, the bot cleared 204 tickets in about 10 minutes with a support engineer reviewing and approving the batch, the company says. The SpaceXSI customer support team uses the Support Bot for customer-facing communication. m:…9314
The pattern across all five threads: agents graduate from pilots when someone measures the trajectory, governs the defaults, and lets the bot own a whole job. m:…4354 m:…9841 m:…2350 m:…5770 m:…9314
What is still unresolved
- Snowflake's eval guidance and x.ai's incident figures are vendor-reported; no independent replication appears in the evidence.
- Spotnana's traveler-facing capabilities and Toyota Finance's PoC are forward-looking; no outcome data exists yet.
- Whether 20–50 golden examples suffice beyond Snowflake's use cases is untested in the evidence.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.