Matrix Derivatives

Common Used in Machine Learning

There are two types of layout in the expression of matrix derivatives: denominator and numberator layout.

Here we always use the denominator layout.

1. vector to vector

$$\frac{\partial \mathbf{A} \mathbf{x}}{\partial \mathbf{x}}=\mathbf{A}^{\top}$$ $$\frac{\partial \mathbf{x}^{\top} \mathbf{A}}{\partial \mathbf{x}}= \mathbf{A}$$

1. scaler to vector

$$\frac{\partial \mathbf{x}^{\top} \mathbf{A} \mathbf{x}}{\partial \mathbf{x}}=\left(\mathbf{A}+\mathbf{A}^{\top}\right) \mathbf{x}$$ $$\frac{\partial \mathbf{x}^{\mathbf{T}} \mathbf{x}}{\partial \mathbf{x}}=2 \mathbf{x}$$

These two conclusions are very popular in the derivative calculation of Machine Learning.

Reference

1. 矩阵求导、几种重要的矩阵及常用的矩阵求导公式