softmax

Forward

y = so f t ma x (x)

x = [x_{1}, \dots, x_{n}], y = [y_{1}, \dots, y_{n}]

则有

y_{i} = \frac{e ^{x_{i}}}{\sum _{j = 1}^{n} e ^{x_{j}}}

Note

相当于对输入取指数后计算各项在所有项中的权重。

为保证数值稳定性，实际应用中有时会在计算前减去最大值：

m = i max x_{i}

y_{i} = \frac{e ^{x_{i} - m}}{\sum _{j = 1}^{n} e ^{x_{j} - m}}

结果数值上完全相等。

分两种情况讨论：

若 $i = j$ ：

\frac{\partial y _{i}}{\partial x _{j}} = \frac{\sum _{k = 1}^{n} e ^{x_{k}} e ^{x_{i}} - e ^{x_{i}} \cdot e ^{x_{i}}}{( \sum _{k = 1}^{n} e ^{x_{k}} ) ^{2}} = y_{i} \cdot \frac{\sum _{k = 1}^{n} e ^{x_{k}} - e ^{x_{i}}}{\sum _{k = 1}^{n} e ^{x_{k}}} = y_{i} (1 - y_{i})

若 $i \neq = j$ ：

\frac{\partial y _{i}}{\partial x _{j}} = - \frac{e ^{x_{i}}}{( \sum _{k = 1}^{n} e ^{x_{k}} ) ^{2}} \cdot e^{x_{j}} = - y_{i} y_{j}

可以合并写作：

\frac{\partial y _{i}}{\partial x _{j}} = y_{i} (δ_{ij} - y_{j})

其中

δ_{ij} = {1, i = j 0, i \neq = j

一般后接交叉熵损失， $L = - \sum p_{i} lo g y_{i}$ ， $p$ 为实际分布或 one-hot 编码，合并 backward，有

\frac{\partial L}{\partial x _{i}} = j \sum \frac{\partial L}{\partial y _{j}} \cdot \frac{\partial y _{j}}{\partial x _{i}} = - j \sum \frac{p _{j}}{y _{j}} \cdot y_{j} (δ_{ij} - y_{i}) = - p_{i} + j \sum p_{j} y_{i} = y_{i} - p_{i}

符合直观，直接是预估分布和实际分布的概率之差