/BlinkDL/ GLU Variants Improve Transformer

Code Link

https://github.com/BlinkDL/RWKV-LM

Description

Gated Linear Units (arXiv:1612. 08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Code: https://github.com/BlinkDL/RWKV-LM

Retrieved

2023/01/21

Stars

1008

TOP