Mixture-of-experts routing

Route tokens to experts with a learned gate; change top-k and capacity and watch load balance, dropped tokens, and active parameters.

Theory and worked examples: Mixture of experts

Mixture-of-experts routing

12 synthetic tokens, a seeded synthetic router. Each token picks its top-k experts; each expert has a fixed number of slots.

token	E0	E1	E2	E3	E4	E5	E6	E7
the	0.08	0.03	0.70	0.30	0.10	0.07	0.09	0.04
cat	0.64	0.18	0.04	0.03	0.36	0.10	0.04	0.06
sat	0.15	0.09	0.05	0.57	0.04	0.43	0.03	0.03
on	0.10	0.05	0.10	0.39	0.02	0.61	0.10	0.13
a	0.58	0.42	0.08	0.19	0.14	0.01	0.02	0.03
warm	0.41	0.02	0.03	0.19	0.06	0.07	0.08	0.59
mat	✕	0.13	0.04	0.05	0.02	0.02	0.36	0.04
and	0.10	0.12	0.05	0.02	0.14	0.57	0.09	0.43
then	✕	0.16	0.01	0.05	0.07	0.15	0.43	0.13
it	✕	0.22	0.48	0.06	0.14	0.01	0.06	0.03
slept	✕	0.06	0.11	✕	0.06	0.13	0.04	0.08
.	✕	0.07	0.04	0.03	0.01	0.02	0.36	0.17

Blue cells are chosen experts with their renormalized gate weight; grey numbers are router probabilities of unchosen experts; ✕ means the expert was full.

Expert load vs capacity

processed dropped (over capacity) capacity