Machine learningOther

Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models with Adaptive Expert Placement

Lazarus, a system for training Sparsely-activated Mixture-of-Experts models, has been introduced, which improves workload balance and recovery probability, outperforming existing systems in frequent node failures and real spot instance trace.