Home

/BlinkDL/ GLU Variants Improve Transformer

Code Link
Description
Gated Linear Units (arXiv:1612. 08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Code: https://github.com/BlinkDL/RWKV-LM
Retrieved
2023/01/21
Stars
1008
TOP