(A postscript: In Fall 2008, as a Courant instructor at NYU math, I taught an upper division undergrad probability course in which I gave myself free license to say things like "covariance is an inner product over a space of mean zero rvs," as a treat. At least a few students appreciated this.)

Googling for this mostly brings up my own notes, which suggests to me that I'm falling prey to idiosyncracies of my own academic upbringing. Pointers to alternate terminology that might lead to more successful searches would also be welcome!

Now, here's the question: the concept of bias-variance decomposition is well known in statistics and machine learning. The concept of quasi-optimality is well known in approximation theory. Putting them together is not deep. Surely there must be a reference?

One can do something similar with the noise term, and this is something I teach in my matrix computations class:

www.cs.cornell.edu/courses/cs62...

For a regularized least squares fit, we can get that 𝔼[c]-c_* is the Moore-Penrose pseudoinverse (with regularization) of a piece of r. And from there, we can show *quasi-optimality* of the squared bias: the version learned from data gives a residual within some factor of the best possible.

Consider a generalized linear model s(X) = w(X) c. There is a c_* that minimizes the expected squared bias term 𝔼[r(X)²] where r(X) = f(X)-w(X) c_*; and we can show (again by orthogonality) that 𝔼[(m(X)-f(X))²] = 𝔼[(w(X) (𝔼[c]-c_*))²] + 𝔼[r(X)²].

Now let X be random, and consider

𝔼[(f(X)-s(X))²] = Var[s(X)] + 𝔼[(m(X)-f(X))²]

where m is the mean of s with respect to the sampling variance (the expectation is just over X).

Now suppose f(x) is a (deterministic) function and s(x) is a predictor (based on a noisy sample). Then

𝔼[(f(x)-s(x))²] = Var[s(X)]+(𝔼[s(X)]-f(x))²

This is the bias-variance decomposition (sometimes people add a term for test-time noise, but I'll drop it here). Same idea.

Also got to have lunch with a former student of mine yesterday. Time flies.

- Asvine just released a new pen! The V800 is very pretty. I don't need more fountain pens, but can still admire.
- I get to check out a CT scanner setup tomorrow! Research reasons, nothing medical.
- The CAM students remain an excellent community. They did tie-dye shirts today.