The Core of Backprop 3: Deriving $\frac{\partial L}{\partial M}$ using Matrix Calculus by Marcus ML Foundations 🔍 ## Previous Lessons in this Series - [The Core of Backprop 2: Deriving $\frac{\partial L}{\partial M}$ using Matrix Calculus](https://finetuneinterview.com/lessons/the-core-of-backprop-2-frac-partial-l-partial-m-using-matrix-calculus) - [The Core of Backprop: Deriving $\frac{\partial L}{\partial M}$](https://www.noteblogdoku.com/blog/core-of-backprop-deriving-dl-dm) This is a sequel lesson to [The Core of Backprop 2 using Matrix Calculus](https://finetuneinterview.com/lessons/the-core-of-backprop-2-frac-partial-l-partial-m-using-matrix-calculus). This derivation is the shortest and simpler using a simple trick. For context on why this is important please read - [Neural Network Implementation from Scratch blog post](https://www.noteblogdoku.com/blog/neural-network-implementation-from-scratch) - [Fitting a Simple Linear Model using Gradient Descent blog post](https://www.noteblogdoku.com/blog/fitting-a-simple-linear-model-using-gradient-descent). The focus of this lesson is to derive $\frac{\partial L}{\partial M}$ from the equations: $$ y = Mx$$ $$ L(y) = \sum_i (g_i - y_i)^2 = (g-y)^T(g-y) $$ Where - $y,g \in \mathbb{R}^m$ our predicted & ground truth data respectively - $x \in \mathbb{R}^n$ our input data - $M \in \mathbb{R}^{m \times n}$ our weight matrix - $L \in \mathbb{R}$ our scalar loss # Derivation The goal is to find $$ \frac{\partial L}{\partial M} \in \mathbb{R}^{n \times m}$$ so that we can update our matrix $M$ using gradient descent. Note that we are using *numerator* form for our derivation, so to update $M$ using gradient descent, we'll have to use the transpose of $\frac{\partial L}{\partial M}$ like so: $$ M' = M - \alpha \frac{\partial L}{\partial M}^T $$ Formally, our objective is to find $\frac{\partial L}{\partial M}$ given: $$ L(y) = ||g-y||^2 \quad y = Mx $$ ### Take the differential of $L(y) = ||g-y||^2$ Given that we have the squared norm of the difference between $g$ and $y$, we can express this as their inner product, i.e.: $$ L(y) = ||g-y||^2 = (g-y)^T (g-y) $$ We are going to take the differential of $L$ using the matrix calculus product rule to find: $$ dL = dy^T(y-g) + (y-g)^T dy $$ Since $dy^T(y-g), (y-g)^T dy \in \mathbb{R}$, we know they are equal to their transpose giving us that $dy^T(y-g) = (y-g)^T dy$. Thus this simplifies to $$ dL = 2(y-g)^T dy $$ ### Take the differential of $y = Mx$ Let's take the differential of $y$, note that $M$ is our variable and our input $x$ is a constant, giving us: $ dy = dM \cdot x + M dx$ Since $x$ is a constant $dx=0$, giving us $ dy = dM \cdot x$ ### Plug $dy = dM \cdot x$ into equation for $dL$ Now let's combine these results: $$ dL = 2(y-g)^T dy \quad dy = dM \cdot x$$ giving us $$ dL = 2(y-g)^T dM\ x$$ Notice that since $dL \in \mathbb{R}$ that $2(y-g)^T dM\ x \in \mathbb{R}$ too. Thus this means that $$ 2(y-g)^T dM\ x = trace(2(y-g)^T dM\ x) $$ Using the trace cycling identity (with example matrices $A$, $B$, and $C$ i.e. $$ trace(ABC) = trace(BCA) = trace(CAB) $$ we have that: $$ trace(2(y-g)^T dM\ x) = trace(2x(y-g)^T dM) \in \mathbb{R}$$ Since we know that $trace(2x(y-g)^T dM) \in \mathbb{R}$, we have that this is equal to itself without its trace, i.e. $$ trace(2x(y-g)^T dM) = 2x(y-g)^T dM $$ Thus this gives us that the equation for $dL$ is $$ dL = 2x(y-g)^T dM $$ Using the matrix calculus identity that $$ df = \langle \frac{\partial f}{\partial x} \rangle dx $$ We can see that $$ dL = 2x(y-g)^T dM \rightarrow dL = \langle 2x(y-g)^T, dM \rangle$$ Implying that $$ \boxed{\frac{\partial L}{\partial M} = 2x(y-g)^T} $$ ### Sanity Check: Matrix Shapes As a quick sanity check, we stated at the beginning that $\frac{\partial L}{\partial M} \in \mathbb{R}^{n \times m}$ (numerator format). To check: $2x(y-g)^T \in \mathbb{R}^{n \times m}$ - $x \in \mathbb{R}^n$ - $y,g \in \mathbb{R}^m$ Thus this is an outer product and we can see that the resultant matrix size is $\mathbb{R}^{n \times m}$ since we are multiplying matrix shapes $\mathbb{R}^{n \times 1} \times \mathbb{R}^{1 \times m} = \mathbb{R}^{n \times m}$ ## Previous Lessons in this Series - [The Core of Backprop 2: Deriving $\frac{\partial L}{\partial M}$ using Matrix Calculus](https://finetuneinterview.com/lessons/the-core-of-backprop-2-frac-partial-l-partial-m-using-matrix-calculus) - [The Core of Backprop: Deriving $\frac{\partial L}{\partial M}$](https://www.noteblogdoku.com/blog/core-of-backprop-deriving-dl-dm) This is a sequel lesson to [The Core of Backprop 2 using Matrix Calculus](https://finetuneinterview.com/lessons/the-core-of-backprop-2-frac-partial-l-partial-m-using-matrix-calculus). This derivation is the shortest and simpler using a simple trick. For context on why this is important please read - [Neural Network Implementation from Scratch blog post](https://www.noteblogdoku.com/blog/neural-network-implementation-from-scratch) - [Fitting a Simple Linear Model using Gradient Descent blog post](https://www.noteblogdoku.com/blog/fitting-a-simple-linear-model-using-gradient-descent). The focus of this lesson is to derive $\frac{\partial L}{\partial M}$ from the equations: $$ y = Mx$$ $$ L(y) = \sum_i (g_i - y_i)^2 = (g-y)^T(g-y) $$ Where - $y,g \in \mathbb{R}^m$ our predicted & ground truth data respectively - $x \in \mathbb{R}^n$ our input data - $M \in \mathbb{R}^{m \times n}$ our weight matrix - $L \in \mathbb{R}$ our scalar loss # Derivation The goal is to find $$ \frac{\partial L}{\partial M} \in \mathbb{R}^{n \times m}$$ so that we can update our matrix $M$ using gradient descent. Note that we are using *numerator* form for our derivation, so to update $M$ using gradient descent, we'll have to use the transpose of $\frac{\partial L}{\partial M}$ like so: $$ M' = M - \alpha \frac{\partial L}{\partial M}^T $$ Formally, our objective is to find $\frac{\partial L}{\partial M}$ given: $$ L(y) = ||g-y||^2 \quad y = Mx $$ ### Take the differential of $L(y) = ||g-y||^2$ Given that we have the squared norm of the difference between $g$ and $y$, we can express this as their inner product, i.e.: $$ L(y) = ||g-y||^2 = (g-y)^T (g-y) $$ We are going to take the differential of $L$ using the matrix calculus product rule to find: $$ dL = dy^T(y-g) + (y-g)^T dy $$ Since $dy^T(y-g), (y-g)^T dy \in \mathbb{R}$, we know they are equal to their transpose giving us that $dy^T(y-g) = (y-g)^T dy$. Thus this simplifies to $$ dL = 2(y-g)^T dy $$ ### Take the differential of $y = Mx$ Let's take the differential of $y$, note that $M$ is our variable and our input $x$ is a constant, giving us: $ dy = dM \cdot x + M dx$ Since $x$ is a constant $dx=0$, giving us $ dy = dM \cdot x$ ### Plug $dy = dM \cdot x$ into equation for $dL$ Now let's combine these results: $$ dL = 2(y-g)^T dy \quad dy = dM \cdot x$$ giving us $$ dL = 2(y-g)^T dM\ x$$ Notice that since $dL \in \mathbb{R}$ that $2(y-g)^T dM\ x \in \mathbb{R}$ too. Thus this means that $$ 2(y-g)^T dM\ x = trace(2(y-g)^T dM\ x) $$ Using the trace cycling identity (with example matrices $A$, $B$, and $C$ i.e. $$ trace(ABC) = trace(BCA) = trace(CAB) $$ we have that: $$ trace(2(y-g)^T dM\ x) = trace(2x(y-g)^T dM) \in \mathbb{R}$$ Since we know that $trace(2x(y-g)^T dM) \in \mathbb{R}$, we have that this is equal to itself without its trace, i.e. $$ trace(2x(y-g)^T dM) = 2x(y-g)^T dM $$ Thus this gives us that the equation for $dL$ is $$ dL = 2x(y-g)^T dM $$ Using the matrix calculus identity that $$ df = \langle \frac{\partial f}{\partial x} \rangle dx $$ We can see that $$ dL = 2x(y-g)^T dM \rightarrow dL = \langle 2x(y-g)^T, dM \rangle$$ Implying that $$ \boxed{\frac{\partial L}{\partial M} = 2x(y-g)^T} $$ ### Sanity Check: Matrix Shapes As a quick sanity check, we stated at the beginning that $\frac{\partial L}{\partial M} \in \mathbb{R}^{n \times m}$ (numerator format). To check: $2x(y-g)^T \in \mathbb{R}^{n \times m}$ - $x \in \mathbb{R}^n$ - $y,g \in \mathbb{R}^m$ Thus this is an outer product and we can see that the resultant matrix size is $\mathbb{R}^{n \times m}$ since we are multiplying matrix shapes $\mathbb{R}^{n \times 1} \times \mathbb{R}^{1 \times m} = \mathbb{R}^{n \times m}$ Comments (0) Please log in to comment. No comments yet. Be the first to comment! ← Back to Lessons
Comments (0)
Please log in to comment.
No comments yet. Be the first to comment!