Backpropagation, step by step

1. 1. Weighted sum$z = w_1x_1 + w_2x_2 + b = 0.5000\cdot2.0000 + (-1.0000)\cdot1.0000 + 0.0000$$z = 0.0000$
2. 2. Sigmoid output$\hat{y} = \sigma(z) = \frac{1}{1 + e^{-0.0000}}$$\hat{y} = 0.5000$
3. 3. Binary cross-entropy$L = -[y\log\hat{y} + (1-y)\log(1-\hat{y})]$$L = 0.6931$
4. 4. Error signal$\frac{\partial L}{\partial z} = \hat{y} - y = 0.5000 - 1$$-0.5000$
5. 5. Gradient for w₁$\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial z}\,x_1 = (-0.5000) \cdot 2.0000$$-1.0000$
6. 6. Gradient for w₂$\frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial z}\,x_2 = (-0.5000) \cdot 1.0000$$-0.5000$
7. 7. Gradient for b$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial z} \cdot 1$$-0.5000$
8. 8. Update (η = 0.1)$w_1 = 0.5000 - 0.1\cdot(-1.0000),\; w_2 = (-1.0000) - 0.1\cdot(-0.5000),\; b = 0.0000 - 0.1\cdot(-0.5000)$$0.6000,\; -0.9500,\; 0.0500$
9. 9. New prediction$z' = 0.6000\cdot2.0000 + (-0.9500)\cdot1.0000 + 0.0500 = 0.3000$$\hat{y}' = 0.5744$

Why is step 4 so simple? $\frac{\partial L}{\partial \hat{y}} = \frac{\hat{y}-y}{\hat{y}(1-\hat{y})}$ and $\frac{\partial \hat{y}}{\partial z} = \hat{y}(1-\hat{y})$. Multiplying them (the chain rule), the $\hat{y}(1-\hat{y})$ terms cancel.