Voice assistants forget how you sounded, because the transcript throws the sound away

The transcript is not the conversation

In a preprint, researchers introduce VoiceLongMemEval, a benchmark for testing whether AI assistants remember paralinguistic metadata, emotion, prosody and voice events, across long-horizon, multi-session conversations. Their central finding is that standard speech-to-text pipelines systematically discard those signals: the words survive into text, and how they were said does not. That creates what the authors call an affect gap. An assistant that remembers what a user said in a previous session, but not that they sounded distressed, hesitant or sarcastic when they said it, is missing information a human listener would have kept without thinking. m:…0094

The preprint quantifies how much that missing information matters. Providing paralinguistic metadata to text-based models yields an accuracy boost of 0.09 to 0.38 on the benchmark, a wide range that suggests the value depends on the task and on which cues are supplied. Audio-native models, which process sound directly rather than through a transcript, outperform blind models in extracting paralinguistic cues, scoring 0.354 to 0.412 against 0.325. Those absolute scores are low, and the authors present them as evidence of a gap rather than a solved problem: even the better approach is far from reliable at remembering how someone sounded across sessions. m:…0094

Recognition itself is moving to the edge

Two further preprints show the other half of the voice story: recognition is getting small enough to run where the user is, rather than in a data centre. One introduces a block-recurrent architecture for speech recognition on resource-constrained edge devices such as wearables, replacing deep parameterised layers with weight-shared recurrent blocks to reduce model size by an order of magnitude. A model with two shared blocks is reported to recover 97.4% of the accuracy of large reference models on Open ASR Leaderboard benchmarks, while weight reuse reduces memory-access overhead and inference energy, and a mixture-of-experts low-rank adaptation method enables user-specific personalisation at lower inference cost than general-purpose systems. m:…4056

The second preprint goes further down the hardware ladder. A 65.7-million-parameter transducer, quantised to 2 and 4 bits with Hadamard-rotated groups and custom SIMD kernels, fits into 26.65 MB of memory and runs in real time on an ESP32-P4 microcontroller, at 0.90 seconds per 1.28-second chunk, without a dedicated neural accelerator. The author reports that GPTQ quantisation is the primary driver of accuracy retention in the 2 and 4-bit process. The point for product teams is not the microcontroller itself. It is that the cost and privacy calculus for voice changes when recognition can run on the device: audio need not leave the room to be transcribed, which matters precisely because, as the first preprint shows, audio carries more than words. m:…3187

Remembering tone is a design choice with privacy attached

Read together, these preprints frame a choice that voice product teams cannot avoid much longer. If an assistant is meant to remember a user across sessions, it must decide what to remember: words only, or words plus how they were said. Keeping paralinguistic signals improves performance on the benchmark, but it also means storing more personal information, for longer, about emotion and voice characteristics. Running recognition on-device reduces one privacy risk, by keeping audio local, while remembering tone across sessions creates another, by keeping a richer profile. The honest product question is not whether voice assistants should remember how you sounded. It is what they remember, for how long, and who can see it, because the transcript alone was never the whole conversation. m:…0094 m:…4056 m:…3187