Tag: evaluation
Truthiness in source evaluation
2026-06-22 truthfulness evaluation text When an LLM assistant evaluates data sources in a social context, it seems to prefer sources with authoritative-sounding methodology markers even if the actual numbers involved do not make sense. Access: $ Basic
LLMs are Predictably Predictable
2026-05-18 alignment evaluation sampling text Some tasks we'd like language models to do, require them to make random selections. Are the models actually able to do that without external help? Access: $ Basic
Moralizing AI opposition
2026-03-16 psychology evaluation meta Who do people oppose AI? Despite that people put forward objections that appear to relate to the consequences of the technology, research suggests that for many opponents, it is really seen as a moral issue. Access: Public
The Well-Actually Test
2026-02-16 alignment evaluation hallucination text tools GPT truthfulness Language models may produce untrue output either by failing to accurately represent training data, or, more insidiously, by accurately representing human misconceptions embedded in the training data. The TruthfulQA benchmark attempts to measure the latter effect. But does it raise insurmountable philosophical problems? Access: Free account
Truthiness-focused search
2026-02-09 LLaMA evaluation hallucination sampling text truthfulness It appears that the earlier, shallower layers of a transformer-type language model learn syntax, and later, deeper layers learn factual information. So can we boost factual accuracy by boosting the effect of deeper layers? I take the view that that's analogous to dosing the model with a mind-altering drug. Access: $$$ Pro
Betting on sycophancy
2026-01-26 evaluation hallucination text truthfulness Chat models have a well-known tendency toward sycophancy: affirming the user's beliefs, even when the user is wrong. But this effect is confounded with several other effects. In this paper the authors attempt to isolate sycophancy by framing questions as a zero-sum game or bet between two humans. Access: $ Basic
The BLEU sausage
2025-12-29 translation evaluation text tools Every paper has an "evaluation" table showing how the paper's new idea gives greater numbers than previous work in the same domain; but where do those numbers actually come from? Here we look at BLEU, a classic measurement for evaluating the quality of machine translation. Access: $ Basic