Re: multicollinearity in regression

There are many excellent statistical experts in this group but let me
give you MY explanation on multicollinearity.

MLR model is
Y=XB
The least square estimator of B is
B-hat=inv(X'X)X'Y

Finding B-hat involves the computation of the INVERSE matrix of X'X.
Computing the inverse matrix is THE main cause of problems in many
areas. Here, X'X is the same as covariance or correlation matrix of X
depending on whether the input columns are normalized or standardized.

When the predictor variables are correlated among themselves,
ill-conditioning, near-singular, or multicollinearity are said to
exist. If the predictor variables are highly correlated, the standard
errors of the estimated partial regression coefficients are so wide
that their interpretation is impossible, therefore the simple
interpretation of the partial regression coefficients as measuring
marginal effects (slope or sensitivity) is unwarranted. If X'X is near
singular, the inverse matrix of X'X are likely to be quite unstable or
possibly even not unique.

For example, let
A=X'X =[1 3 4
3 9 12
4 12 16]
The rank, the number of independent columns or rows, of matrix A = 1,
since the second and the third columns are obtained from the first
column by multiplying 3 and 4. Because the A is singular, the inverse
matrix, A-1 does not exist.

However, let's add small random noises to A and called it A1
A1=[1.02 3.08 4.05
2.95 9.01 12.01
4.06 12.01 15.99]

Inverse(A1) =
[ -2.7474 -9.8258 8.0760
25.6679 -2.1502 -4.8863
-18.5814 4.1098 1.6820]

Another matrix with small random noises added;
A2=[1.01 3.03 4.02
2.97 9.00 12.05
4.01 11.59 15.97]

Inverse (A2) =
[ 39.6971 -17.5280 3.2329
8.6757 0.0926 -2.2538
-16.2641 4.3340 0.8865]

Therefore, adding small random noises cause the two inverse matrices
change totally. Let me state clearly that multicollinearity does not
necessarily prevent MLR from satisfactory prediction IF the new values
are like the ones used to develop the prediction model. How do we know
the new samples are like the ones used to develop the prediction model?
My approach is to compute the 95% confidence ellipse based on the
principal component scores.

As an analogy to find the least square estimation, let's Y=10X1+5X2.
The partial derevative of X1 (əY/əX1) is 10, right? ...... Not quite
right because, taking the partial derivative of X1 already assumes that
the variable X2 are held CONSTANT. If X1 and X2 are collinear, there
is no way to hold other variable held constant, thus the least square
estimator approach automatically assume that the input variables are
orthogonal.

Multicollinearity is an ubiquitous problem affecting the computation of
inverse matrix. Artificial neural networks is not free from the
multicollinearity since the chain rule of partial derivatives is used
in order to derive the weights, where the partial derevative assumes
that other variables are hold constant and there is no way from
observational data that the other variables are hold constant.

What's the solution to the ill-conditioning? Dozens of methods have
been developed such as LU decomposition, QR decomposition,
eigenvalue/vector decomposition (EVD), singular value decomposition
(SVD), ridge regression. My approach (which is not perfect though) is
using PCA (principal component analysis), PCR (principal component
regression), PLS (partial least square).

Sorry I can't give my explanation to your original query since much
information is missing......observational or experimental data,
objective, etc.

Hope this helps.

Sangdon Lee, Ph.D.,
GM Tech. Center.

.