Information Theory

Derivation of $\frac{\partial L}{\partial M}$ for Cross Entropy Loss $L(y) = - \sum_{i=1}^C g\ log(y)$ with ground truth $g$, predicted $y$

For Probabilistic Classifiers, MLE Reduces to Cross-Entropy Loss

$\nabla D_{KL}(P||Q) = \nabla H(P,Q)$, $D_{KL}(P||Q)$ vs $D_{KL}(Q||P)$, & Intuitive Visualizations

Connecting Self-Information & Entropy for Distribution Sampling & Prefix Encoding