Teddy Bloggie Blog Blogging

From Argmax

When you arrive at the Pudong International Airport in Shanghai, skip the taxis... take the maglev train. Closest feeling I've ever had to warp speed.

Date: 3/31/2004

I learned a while ago that the ordinary least square (OLS) variance is =sigma^2/(X'X), but I didn't really know why intuitively until today.

In statictics we're always walking in a valley made of of data. Our only job is to look for the low spot in the valley by picking the right betas (B's), betas are the coordinates that describe our location. If the valley is nicely bowl shaped, we should just curl into a ball and let gravity take us there. Well, we have computers, so we can't just curl into a ball, we have to stand upright like robots and take little steps. It's still relatively easy for a computer to find the low spots when it's nicely shaped. Sometimes the valley is flatter and we can't discern the gentle slope that led here from the mountains in the distance, and so computers find it hard to find the lowest spot also.

In OLS, the criterion is minimize sum of square errors (E'E). Pick our betas, look around and see if we're at the bottom of the valley in terms of E'E. In calculus, we would take a derivative and see if it's zero.

Since E'E=(Y-XB)'(Y-XB)

The first derivative of E'E is = 2/(X'X)*B-2X'Y
Setting the result to zero and solving for B gets you to the min.

But the second derivative (the Hessian) of E'E = 1/(X'X) tells you the variance. Well, actually, the variance is sigma^2/(X'X) but this is "relationship" not "equivalence". The relationship here is between how certain you can be about beta given the criterion and the data; not how big a role the real error plays in the variance of beta. In any case, the Hessian determines how big a role sigma squared has to play. If the Hessian is large, the variance is going to be small, and if the Hessian is small, the variance is going to be relatively larger.

In calculus terms, the first derivative when set to zero and solved tells you that you're at the minimum of the criterion (E'E), so you've found the flat spot in the criterion to a first order. But the second derivative at the point of the solution gives you the variance/covariance matrix in least squares -- it tells you, as you've walked to the min of the criterion valley, how steeply the path around you appear. If the path was steep, you're pretty sure you're at the min. If the valley floor is pretty flat and changes slowly, so that (X'X) is a small number, then your variance is going to be large since you're not going to be sure that you're at the absolute lowest point in the valley you've just explored.

This isn't strictly correct mathematically because we're in a K dimension valley usually, but I think it gives me something to hang onto when I'm writing (X'X) and trudging through another econometrics class.