The factuality reckoning: alignment rewards format, not truth

The alignment methods don't beat the base model

A reproducibility study took five major factuality alignment methods — PPO, GRPO, Dr. GRPO, DPO, KTO — re-ran them independently, and found they "fail to consistently improve factuality". often doing no better than the base model on a common benchmark. The diagnosis is damning: the alignment rewards "prioritize format, length, or brevity over factual relevance". The model learns to sound right rather than be right. m:…4779

The measurement side is no better. The authors argue current factuality benchmarks are "fragile due to the lack of ground truth and reliance on LLM judges" — we're grading truthfulness with systems that have the same failure mode. If your eval can't tell confident from correct, your training can't either. m:…4779

This suggests a uncomfortable possibility for anyone buying factuality-tuned models: the tuning may be optimizing the wrapper, not the knowledge. The paper's implication is that the field needs different benchmarks before it needs different methods. m:…4779

The five reproduced methods span the standard toolkit — PPO, GRPO, Dr. GRPO, DPO, KTO — which makes the null result harder to dismiss as a bad implementation. These are the techniques the industry actually uses to make models more truthful, and under independent reproduction they "do not significantly outperform base models on a common benchmark." The failure mode the authors identify — rewards that key on length, formatting and brevity — is Goodhart's law wearing a lab coat. m:…4779

Two different attacks on the real problem

SPARK starts from a specific, well-known RAG failure: the model trusts its own parametric memory over the retrieved evidence in front of it. Rather than full fine-tuning, SPARK adds a lightweight latent participation policy that modulates FFN-mediated knowledge — just 0.01% additional parameters, backbone fully frozen, general capabilities preserved. It's a surgical fix for the model-knows-best problem. m:…3703

EvoRAG attacks from the knowledge side. Traditional knowledge-graph RAG is static; EvoRAG closes the loop with feedback backpropagation, attributing response-level feedback down to individual triplets so the graph refines itself toward what actually helps. The reported gain: 7.34% better reasoning accuracy than state-of-the-art KG-RAG frameworks. m:…6789

Both papers share a philosophy the alignment study implicitly endorses: stop trying to make the model more truthful in the abstract, and fix the concrete mechanism — which knowledge wins a conflict, which triples survive feedback. Factuality, in this view, is an engineering property of the system, not a virtue of the weights. m:…3703 m:…6789

SPARK's mechanism deserves a closer look because it's deliberately minimal. Rather than retraining the model to be faithful, it learns a latent participation policy that dials the FFN-mediated parametric knowledge up or down depending on the retrieved context — the model keeps everything it knows, but learns when to defer to the document. At 0.01% additional parameters, it's closer to a steering wheel than an engine rebuild. m:…3703

EvoRAG's feedback loop is the mirror image: instead of steering the model, evolve the knowledge. By attributing response-level feedback to individual triplets and filtering low-contribution knowledge, the graph becomes a living artifact that adapts to the task — a closed loop where the retrieval substrate improves with use rather than rotting from the day it's built. m:…6789