{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "882",
  "slug": "voice-assistants-forget-how-you-sounded",
  "edition": "2026-10-31T00",
  "url": "https://hangingcontext.com/stories/2026-10-31/voice-assistants-forget-how-you-sounded/",
  "newsroom": "artificial-intelligence",
  "beat": "models-labs",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Voice assistants lose how a person sounded when speech is turned into text, and the fix is audio-native memory plus on-device recognition.",
  "headline": "Voice assistants forget how you sounded, because the transcript throws the sound away",
  "dek": "Preprints published October 30 show that standard speech pipelines discard emotion, prosody and voice events, that audio-native models recover some of that signal, and that capable speech recognition now runs on a microcontroller.",
  "published_at": "2026-10-31T00:37:47Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 3 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 3,
  "publishers": [
    "arxiv-ai-infra-inference-ops",
    "arxiv-llm-evals-benchmarks",
    "arxiv-model-efficiency-engineering"
  ],
  "key_facts": [
    {
      "text": "In a preprint, standard speech-to-text pipelines systematically discard paralinguistic signals such as emotion, prosody and voice events.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793346404577730094",
          "claim_ids": [
            "1793346826514818865"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, giving text-based models paralinguistic metadata lifts accuracy by 0.09 to 0.38 on a long-horizon voice memory benchmark.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793346404577730094",
          "claim_ids": [
            "1793346826521936894"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, audio-native models score 0.354 to 0.412 on paralinguistic cues against 0.325 for blind models.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793346404577730094",
          "claim_ids": [
            "1793346826527492110"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, a block-recurrent speech model with two shared blocks recovers 97.4% of the accuracy of large reference models while cutting model size by an order of magnitude.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793360486738224056",
          "claim_ids": [
            "1793360718298592790",
            "1793360718300308661"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, a 65.7-million-parameter speech transducer quantised to 2 and 4 bits fits in 26.65 MB and runs in real time on an ESP32-P4 microcontroller without a neural accelerator.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793359933822673187",
          "claim_ids": [
            "1793360197539562656",
            "1793360197543300011"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The transcript is not the conversation",
      "paragraphs": [
        {
          "markdown": "In a preprint, researchers introduce VoiceLongMemEval, a benchmark for testing whether AI assistants remember paralinguistic metadata, emotion, prosody and voice events, across long-horizon, multi-session conversations. Their central finding is that standard speech-to-text pipelines systematically discard those signals: the words survive into text, and how they were said does not. That creates what the authors call an affect gap. An assistant that remembers what a user said in a previous session, but not that they sounded distressed, hesitant or sarcastic when they said it, is missing information a human listener would have kept without thinking.",
          "cites": [
            {
              "manifest_id": "1793346404577730094",
              "claim_ids": [
                "1793346826514818865"
              ]
            }
          ]
        },
        {
          "markdown": "The preprint quantifies how much that missing information matters. Providing paralinguistic metadata to text-based models yields an accuracy boost of 0.09 to 0.38 on the benchmark, a wide range that suggests the value depends on the task and on which cues are supplied. Audio-native models, which process sound directly rather than through a transcript, outperform blind models in extracting paralinguistic cues, scoring 0.354 to 0.412 against 0.325. Those absolute scores are low, and the authors present them as evidence of a gap rather than a solved problem: even the better approach is far from reliable at remembering how someone sounded across sessions.",
          "cites": [
            {
              "manifest_id": "1793346404577730094",
              "claim_ids": [
                "1793346826521936894",
                "1793346826527492110"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Recognition itself is moving to the edge",
      "paragraphs": [
        {
          "markdown": "Two further preprints show the other half of the voice story: recognition is getting small enough to run where the user is, rather than in a data centre. One introduces a block-recurrent architecture for speech recognition on resource-constrained edge devices such as wearables, replacing deep parameterised layers with weight-shared recurrent blocks to reduce model size by an order of magnitude. A model with two shared blocks is reported to recover 97.4% of the accuracy of large reference models on Open ASR Leaderboard benchmarks, while weight reuse reduces memory-access overhead and inference energy, and a mixture-of-experts low-rank adaptation method enables user-specific personalisation at lower inference cost than general-purpose systems.",
          "cites": [
            {
              "manifest_id": "1793360486738224056",
              "claim_ids": [
                "1793360718298592790",
                "1793360718300308661",
                "1793360718302596665",
                "1793360718304512107"
              ]
            }
          ]
        },
        {
          "markdown": "The second preprint goes further down the hardware ladder. A 65.7-million-parameter transducer, quantised to 2 and 4 bits with Hadamard-rotated groups and custom SIMD kernels, fits into 26.65 MB of memory and runs in real time on an ESP32-P4 microcontroller, at 0.90 seconds per 1.28-second chunk, without a dedicated neural accelerator. The author reports that GPTQ quantisation is the primary driver of accuracy retention in the 2 and 4-bit process. The point for product teams is not the microcontroller itself. It is that the cost and privacy calculus for voice changes when recognition can run on the device: audio need not leave the room to be transcribed, which matters precisely because, as the first preprint shows, audio carries more than words.",
          "cites": [
            {
              "manifest_id": "1793359933822673187",
              "claim_ids": [
                "1793360197539562656",
                "1793360197543300011",
                "1793360197551378659"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Remembering tone is a design choice with privacy attached",
      "paragraphs": [
        {
          "markdown": "Read together, these preprints frame a choice that voice product teams cannot avoid much longer. If an assistant is meant to remember a user across sessions, it must decide what to remember: words only, or words plus how they were said. Keeping paralinguistic signals improves performance on the benchmark, but it also means storing more personal information, for longer, about emotion and voice characteristics. Running recognition on-device reduces one privacy risk, by keeping audio local, while remembering tone across sessions creates another, by keeping a richer profile. The honest product question is not whether voice assistants should remember how you sounded. It is what they remember, for how long, and who can see it, because the transcript alone was never the whole conversation.",
          "cites": [
            {
              "manifest_id": "1793346404577730094",
              "claim_ids": [
                "1793346826514818865",
                "1793346826521936894"
              ]
            },
            {
              "manifest_id": "1793360486738224056",
              "claim_ids": [
                "1793360718304512107"
              ]
            },
            {
              "manifest_id": "1793359933822673187",
              "claim_ids": [
                "1793360197543300011"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Whether paralinguistic memory helps in real conversations, or only on the benchmark's structured multi-session tasks.",
    "How long paralinguistic profiles should be retained, and whether users can inspect or delete them, which the preprints do not address.",
    "Whether on-device recognition accuracy holds in noisy real rooms, rather than in the tested benchmark and microcontroller conditions."
  ],
  "decision_card": {
    "useful_for": [
      "Developers building voice assistants or wearables",
      "Product teams deciding what voice data to retain",
      "Edge AI teams deploying speech on constrained hardware"
    ],
    "not_for": [
      "Teams building text-only chat products",
      "Anyone needing a privacy compliance ruling"
    ],
    "compare_against": [
      "Text-only pipelines that discard audio cues",
      "Cloud speech recognition for every utterance",
      "General-purpose speech models without personalisation"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-31T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793346404577730094",
    "1793360486738224056",
    "1793359933822673187"
  ],
  "manifests": [
    {
      "manifest_id": "1793346404577730094",
      "title": "VoiceLongMemEval: Do Assistants Remember How You Sounded?",
      "source_name": "arxiv-llm-evals-benchmarks",
      "source_url": "https://arxiv.org/pdf/2609.00570v3",
      "published_date": "2026-10-30"
    },
    {
      "manifest_id": "1793360486738224056",
      "title": "Depth through Recurrence: Towards Ultra-Efficient On-Device ASR",
      "source_name": "arxiv-model-efficiency-engineering",
      "source_url": "https://arxiv.org/pdf/2610.37833v1",
      "published_date": "2026-10-30"
    },
    {
      "manifest_id": "1793359933822673187",
      "title": "Streaming Speech Recognition on a Microcontroller: a 65.7M-Parameter Transducer in 2 and 4 Bits on the ESP32-P4",
      "source_name": "arxiv-ai-infra-inference-ops",
      "source_url": "https://arxiv.org/pdf/2610.38739v1",
      "published_date": "2026-10-30"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793346404577730094",
      "1793360486738224056",
      "1793359933822673187"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793346404577730094",
      "all": "for id in 1793346404577730094 1793360486738224056 1793359933822673187; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-31/voice-assistants-forget-how-you-sounded/",
    "json": "https://hangingcontext.com/stories/2026-10-31/voice-assistants-forget-how-you-sounded/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-31/voice-assistants-forget-how-you-sounded/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-31T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-31/training-budgets-are-search-budgets/",
      "headline": "Training cost is now search cost: configs, rollouts, data mixes and optimisers all get cheaper searches",
      "slug": "training-budgets-are-search-budgets"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-31/computer-agents-act-on-stale-screens/",
      "headline": "The computer-use agent clicked too early: stale screens, moving goals and false finishes",
      "slug": "computer-agents-act-on-stale-screens"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-31/synthetic-data-blind-spots/",
      "headline": "Synthetic users are fluent, confident, and wrong about the parts of life the internet never wrote down",
      "slug": "synthetic-data-blind-spots"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-31/inference-serving-squeezes-seconds/",
      "headline": "Serving is the new model race: Baseten and NVIDIA squeeze seconds out of weight syncs and storage",
      "slug": "inference-serving-squeezes-seconds"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Claims with a source link",
    "text": "A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it.",
    "cta": {
      "label": "Read the guide",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-31/voice-assistants-forget-how-you-sounded/"
  }
}