Large language models systematically produce outputs with lower diversity than their training data, according to new research published on arXiv. The study examined whether LLMs capture the full range of plausible outputs present in their training datasets.
According to the research paper, the team measured output variability using conditional entropy and matrix-based von Neumann entropy. Testing across LLM families with publicly available training data—including OLMo, Pythia, and GPT-Neo—researchers “consistently find that model-generated outputs exhibit lower conditional entropy than their training data, across different model scales, sequence lengths, and decoding strategies,” the paper states.
This “conditional diversity gap” extends beyond language modeling, with similar patterns observed in class-conditioned ImageNet generators and text-conditioned models trained on MS-COCO, according to the study.
To address this limitation, the researchers proposed a post-hoc correction mechanism that generates multiple outputs for each input and reweights them through a matrix-entropy projection. According to the paper, this approach increases conditional diversity while remaining close to the original model distribution. The team developed a scalable mirror-descent algorithm for implementation, proving the concavity of the matrix-based conditional entropy functional.
The research provides “an information-theoretic framework for measuring and mitigating this gap” between modern generative models and their training data, according to the authors.