MoE for Mixture of Experts.

MoE vs Dense

Traditional Transformer:

But need most of the memory and calculation.

  • Insight: Perhaps it’s redundant? Perhaps we should let tokens choose what to calculate?
  • Why not split it into multiple experts?
  • MoE models are sparse models.

Each token pick some experts to calculate. Use a router to determine which experts to use.

Expert Collapse

Initially, every experts work well. After some steps, several experts might stand out. Here’s the problem: the model finds out these experts work better, so the router sends more tokens to these experts, and they get more training, and work better and better. A vicious cycle. Finally, tokens are only send to these experts, which is called Expert Collapse.

Load Balancing Loss

The solution is adding a Load Balancing Loss.

We want each expert gets nearly equal number of tokens.

For expert ,

determines average probability for tokens to go to expert , and

determines how many tokens are actually assigned to expert .

Intuitively, we are trying to lower the loss, thus increasing the probability to assign tokens to those experts receive less tokens and vice versa.

However, we are modifying the routing model want to learn, which might be harmful.

Capacity

Model cannot really evenly routes the tokens. But the GPU cannot allocate infinite spaces for each experts either. Therefore, we assign a capacity for experts, and drop exceeding tokens.

System Perspective

The idea and algorithm is quite easy:

for token in tokens:
	experts = router(token)
	for e in experts:
		output += e(token)

However, you will never want to implement this way, since the performance will be really bad.

We need to permute tokens to make them grouped by the experts they are routed to. Then we send them to the experts in a batch manner and perform Grouped GEMM.

Expert Parallelism

So many experts cannot be put into one GPU. When we seperate the experts, we will have to send tokens to coresponding GPUs, and the results have to be sent back to original GPU. Therefore, the bottleneck of MoE is always not computation, but communication.

Grouped GEMM

Experts on one GPU might receive different amount of tokens, and therefore we can’t calculate with normal batch GEMM. Grouped GEMM is used to solve the problem: we launch a single kernel and it tackles several GEMMs with different shapes.

Variants

Shared Expert

Sometimes we want all tokens get through some shared experts, and then get through different routed experts.

Fine-grained Experts

We don’t use few giant experts, but many tiny experts. For example, 8 experts, 14336 intermediate size, top-2 vs 64 experts, 1792 intermediate size, top-16.

The benefit is that the number of possible routing combinations greatly increases from to . But the problem is there will be smaller GEMMs and more communication fragmentation.