Quizzes
Scaling and distributed training
Covers the lesson Scaling and distributed training.
Quiz: Scaling and distributed training
Question 1 of 10
Calculationeasy
With mixed-precision AdamW (bf16 weights and gradients, fp32 master copy and two fp32 Adam moments), how many bytes per parameter does training state take, before activations?