Maximum likelihood notes

This is a public fixture post (PUBLIC_KATEX_FIXTURE) used to check KaTeX, code overflow, and public-build inclusion.

Let $x_1,\ldots,x_n \in \R^d$ be i.i.d. samples from $\mathcal{N}(\mu,\sigma^2 I)$. The negative log-likelihood is

$$\ell(\mu,\sigma^2) = \frac{nd}{2}\log(2\pi\sigma^2) + \frac{1}{2\sigma^2}\sum_{i=1}^n \norm{x_i - \mu}^2.$$

The MLE is the usual mean, $\hat\mu = \argmax_{\mu} \E[\log p_\mu(x)]$, which here reduces to $\hat\mu = n^{-1}\sum_i x_i$. For two densities $p$ and $q$, the gap between them is the KL divergence $\kl(p\|q)$.

A useful scalar identity in one dimension is $\Var[X] = \E[X^2] - \E[X]^2$, with $\P(X \in A) = \E[\ind_{A}(X)]$.

import numpy as np

def gaussian_nll(x, mu, sigma2):
    residual = x - mu
    return 0.5 * (np.log(2 * np.pi * sigma2) + (residual ** 2) / sigma2)

KaTeX is not TeX Live: macros such as \R and \norm are defined in src/lib/katex_macros.ts. Diagrams that need TikZ should be pre-rendered.