Demo: Linear Models and Least Squares
View ordinary least squares as projection onto a one-dimensional column space, then compare it with the geometry induced by weighted least squares.
Mathematical setup
The toy model is $y=h\theta+e$ with
\[h= \begin{bmatrix} 1\\ \text{slope} \end{bmatrix}.\]Ordinary least squares solves
\[\hat\theta_{\mathrm{OLS}}=(h^Th)^{-1}h^Ty, \qquad h^T(y-h\hat\theta_{\mathrm{OLS}})=0.\]Weighted least squares with $W=\operatorname{diag}(1,w_2)$ solves
\[\hat\theta_{\mathrm{WLS}}=(h^TWh)^{-1}h^TWy, \qquad h^TW(y-h\hat\theta_{\mathrm{WLS}})=0.\]The weighted residual is orthogonal in the $W$-inner product, so it need not be Euclidean-orthogonal to the column space.
What to try
- Set $w_2=1$. OLS and WLS coincide because the two geometries are the same.
- Increase $w_2$. The fitted point moves to respect the second coordinate more strongly.
- Change the slope. The column space rotates, and both projections change even when the observed $y$ is fixed.
For weighted least squares or GLS after whitening, the normal equation becomes $H^T W(y-H\hat\theta)=0$. The unweighted residual dot product need not be zero for the weighted fit.