Notes on Transformers
Various notes on Transformers.
Mixture of Experts
- Helps scale model size without increasing FLOPs
- Dipatch to expert works with
- Learned routing (Switch Transformers)
- Randomly (with capacity cap, GShard)
- Hashing (baseline, not learned)
- HF blog post on MoE.
- MoE is typically applie to MLP layers but can also be applied to attention heads.