-
Welford's Algorithm: The Hidden Numerical Trick in LayerNorm
When to use Welford's Algorithm?
-
NCCL Primitives Deep Dive
Understanding All-Gather, Ring All-Reduce, and All-to-All communication primitives in LLMs
When to use Welford's Algorithm?
Understanding All-Gather, Ring All-Reduce, and All-to-All communication primitives in LLMs