Machine learningOther

u-μP: The Unit-Scaled Maximal Update Parametrization

The u-$\mu$P scheme merges Maximal Update Parametrization and Unit Scaling techniques to make model hyperparameters size-independent and easier to train in low-precision, leading to more efficient models that function immediately in FP8.