Shae's Ramblings

In this post, we aim to prove that any continuous density \(x\mapsto p(x)\) over a bounded subset \(\mathcal{X} \subset [a,b]\) can be approximated by a convex combination of deltas \(\delta_{x_i}\): there is always a \(N\) for which we have with \(\varepsilon\)-accuracy that \(p(x) \approx p_n(x) = \sum_{i=1}^n\gamma_i \delta_{x_i}(x)\) for all \(n > N\).

Because any bounded subset in \(\mathbb{R}\) can be finitely covered by closed intervals \([a_i, b_i]\), we can prove our result for each interval and use the fact that for any \(c,d\in\mathcal{X}\) bounded, the integral is equal to

$$ \int_c^dp(x)\mathrm{d}x = \sum_{i=1}^n\int_{a_i}^{b_i}p(x)\mathrm{d}x $$

Thus to simplify the problem, we can assume \(\mathcal{X} = [a,b]\) some interval.

Weak convergence

For continuous distributions \(f,g\) on \(\mathcal{X}\) define the inner product

$$ \langle f,g\rangle \triangleq \int_\mathcal{X}f(x)g(x)\,\mathrm{d}x $$ We say a sequence \(f_n\) converges weakly to a limit \(f_\star\) if for every distribution \(g\) on \(\mathcal{X}\) we have that \(\langle f_n, g\rangle \to \langle f_\star, g\rangle\) as \(n\to\infty\).

The proof

We can use Riemann sums to prove the statement. First, notice that because \(\int p(x)\mathrm{d}x = 1\), we can for any \(n\) find a partition \(a = x_0 < x_1 < \dots < x_n\) with gaps \(\Delta x_i = x_{i+1} – x_i\) so that

$$ \sum_{i=0}^{n-1}p(x_i^\star)\Delta x_i = 1 + u_n $$ with \(u_n \to 0\) as \(n\) tends to infinity. We can assume that \(u_n \leq 0\) by taking \(p(x_i^\star) = \inf\{p(x)\mid x_i \leq x \leq x_{i+1}\}\) which is attained on a closed interval by the continuity of \(p\). We can therefore for any \(n\) construct the following \(\gamma_i\):

$$ \gamma_i \triangleq p(x_i^\star)\Delta x_i – u_n/n $$ so that we always have \(\gamma_i \geq 0\) and \(\sum_{i=0}^{n-1}\gamma_i = 1 + u_n – u_n = 1\).

Finally, consider the convex sum \(p_n = \sum_{i=0}^{n-1} \gamma_i \delta_{x_i^\star}\), we have

$$ \langle p_n , f\rangle = \sum_{i=0}^{n-1} \gamma_i f(x_i^\star) = \sum_{i=0}^{n-1}\left( p(x_i^\star)\Delta x_i – u_n/n \right) f(x_i^\star) $$ which expands into

$$ \sum_{i=0}^{n-1} \gamma_i g(x_i^\star) = \sum_{i=0}^{n-1}p(x_i^\star)f(x_i^\star)\Delta x_i – \dfrac{u_n}{n}\sum_{i=0}^{n-1}f(x_i^\star) $$ which converges to \(\langle p, f \rangle\) because \(\dfrac{u_n}{n}\sum_{i=0}^{n-1} f(x_i^\star) \to 0\) from the bounded nature of \(f\).