Machine learningLLMs & Text

Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Data

The study introduces a measure called the diversity coefficient to formalize data quality in pre-training Large Language Models (LLMs), demonstrating its alignment with diversity and variability properties, and its usefulness in evaluating downstream model performance.