Tag: truthfulness
Truthiness in source evaluation
2026-06-22 truthfulness evaluation text When an LLM assistant evaluates data sources in a social context, it seems to prefer sources with authoritative-sounding methodology markers even if the actual numbers involved do not make sense. Access: $ Basic
Call the Science Police
2026-05-25 alignment sampling text toxicity truthfulness Proposal to improve the scientific accuracy of LLM output in domains like medicine, by using a larger model to write executable rules that are applied to a smaller model's output at search time. Access: $ Basic
The Well-Actually Test
2026-02-16 alignment evaluation hallucination text tools GPT truthfulness Language models may produce untrue output either by failing to accurately represent training data, or, more insidiously, by accurately representing human misconceptions embedded in the training data. The TruthfulQA benchmark attempts to measure the latter effect. But does it raise insurmountable philosophical problems? Access: Free account
Truthiness-focused search
2026-02-09 LLaMA evaluation hallucination sampling text truthfulness It appears that the earlier, shallower layers of a transformer-type language model learn syntax, and later, deeper layers learn factual information. So can we boost factual accuracy by boosting the effect of deeper layers? I take the view that that's analogous to dosing the model with a mind-altering drug. Access: $$$ Pro
Betting on sycophancy
2026-01-26 evaluation hallucination text truthfulness Chat models have a well-known tendency toward sycophancy: affirming the user's beliefs, even when the user is wrong. But this effect is confounded with several other effects. In this paper the authors attempt to isolate sycophancy by framing questions as a zero-sum game or bet between two humans. Access: $ Basic
Quantization and truthfulness
2026-01-05 quantization basics hallucination logic text truthfulness Quantization is rounding off, an important class of techniques for saving space and computation in the use of machine learning models. As well as reviewing the general topic of quantization and floating-point numbers, I discuss experiments on the question of how quantization affects truthfulness, the factual accuracy of answers returned by quantized language models. Access: $ Basic