Machine learningLLMs & Text

Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism

The article introduces a new adaptive batch size schedule for large-scale model training, which optimizes memory usage and performs better than constant batch sizes, especially in pretraining smaller models.