Re: Linear regression vs. vectors from principal components analysis

They are, indeed, intended for very different purposes.

Here's something to consider. If you run a PCA on a covariance
matrix... and then change one of the metrics (I'll explain) you'll get
a different "answer". Yes, the eigenvalues (and of course the
eigenvectors) will change.

Example: One of your variables is recorded in yards. Run the PCA. Now
convert the variable "in yard" to the metric "inches". Run the PCA
again. Oooops!!! The eigenvalues changed. The "answers" depend on the
metrics.

Example: Scale the variables by centering and divide by the respective
standard deviations.
Effectively... run the PCA on the correlation matrix. This produces a
dramatically different "answer".

Bottom line: Putting meaning/interpretation on the eigenvalues and
eigenvectors is a slippery thing.

But nonetheless, PCA is in no way a substitute for regression. OMU


Mitch wrote:
Here's an understandably extremely loaded question:

Why isn't PCA used rather than linear regression?

Let me explain (so that this doesn't seem so contentious, or just plain
'not-even-wrong').

Both methods start with the covariance matrix.

- For regression, you solve a linear system based on that matrix.This
method is asymmetric in the variables, that is, you get different
regression lines depending on which variable you consider the
'independent' variable (this is the source of 'regression to the mean '
as an artifact, not of the data but of the process)
- For PCA, you compute the eigenvalues and eigenvectors of the
covariance matrix. all variables are treated symmetrically. The vector
with the largest eigenvalue could be interpreted as the line affording
the greatest predictive value (the shortest vector affording the least
variability).

Does that comparison make sense? Does PCA really come out of that
better? The asymmetry has started to bother me, but I've never heard it
addressed anywhere.

Mitch

.

Relevant Pages