Mixture of experts (MoE) is a machine learning technique where multiple expert networks (learners) are used to divide a problem space into homogeneous regions. MoE represents a form of ensemble learning. They were also called committee machines.

MoE always has the following components, but they are implemented and combined differently according to the problem being solved:

Both the experts and the weighting function are trained by minimizing some loss function, generally via gradient descent. There is much freedom in choosing the precise form of experts, the weighting function, and the loss function.

The meta-pi network, reported by Hampshire and Waibel, uses ${\displaystyle f(x)=\sum _{i}w(x)_{i}f_{i}(x)}$ as the output. The model is trained by performing gradient descent on the mean-squared error loss ${\displaystyle L:={\frac {1}{N}}\sum _{k}\|y_{k}-f(x_{k})\|^{2}}$. The experts may be arbitrary functions.

In their original publication, they were solving the problem of classifying phonemes in speech signal from 6 different Japanese speakers, 2 females and 4 males. They trained 6 experts, each being a "time-delayed neural network" (essentially a multilayered convolution network over the mel spectrogram). They found that the resulting mixture of experts dedicated 5 experts for 5 of the speakers, but the 6th (male) speaker does not have a dedicated expert, instead his voice was classified by a linear combination of the experts for the other 3 male speakers.

The adaptive mixtures of local experts uses a Gaussian mixture model. Each expert simply predicts a Gaussian distribution, and totally ignores the input. Specifically, the ${\displaystyle i}$-th expert predicts that the output is ${\displaystyle y\sim N(\mu _{i},I)}$, where ${\displaystyle \mu _{i}}$ is a learnable parameter. The weighting function is a linear-softmax function:$${\displaystyle w(x)_{i}={\frac {e^{k_{i}^{T}x+b_{i}}}{\sum _{j}e^{k_{j}^{T}x+b_{j}}}}}$$The mixture of experts predict that the output is distributed according to the log-probability density function:$${\displaystyle \ln f_{\theta }(y|x)=\ln \left[\sum _{i}{\frac {e^{k_{i}^{T}x+b_{i}}}{\sum _{j}e^{k_{j}^{T}x+b_{j}}}}N(y|\mu _{i},I)\right]=\ln \left[(2\pi )^{-d/2}\sum _{i}{\frac {e^{k_{i}^{T}x+b_{i}}}{\sum _{j}e^{k_{j}^{T}x+b_{j}}}}e^{-{\frac {1}{2}}\|y-\mu _{i}\|^{2}}\right]}$$It is trained by maximal likelihood estimation, that is, gradient ascent on ${\displaystyle f(y|x)}$. The gradient for the ${\displaystyle i}$-th expert is

$${\displaystyle \nabla _{\mu _{i}}\ln f_{\theta }(y|x)={\frac {w(x)_{i}N(y|\mu _{i},I)}{\sum _{j}w(x)_{j}N(y|\mu _{j},I)}}\;(y-\mu _{i})}$$

and the gradient for the weighting function is$${\displaystyle \nabla _{[k_{i},b_{i}]}\ln f_{\theta }(y|x)={\begin{bmatrix}x\\1\end{bmatrix}}{\frac {w(x)_{i}}{\sum _{j}w(x)_{j}N(y|\mu _{j},I)}}(f_{i}(x)-f_{\theta }(y|x))}$$