Re: Covariance Mean subtraction in Principal Component Analysis


John D'Errico wrote:
> In article <1126828640.587128.150310@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>,
> "Rune Allnor" <allnor@xxxxxxxxxxxx> wrote:
>
> > fearry wrote:
> > > I was just wondering what is the advantage of subtracting the Mean from the
> > > covariance matrix when computing PCA. Is it valid to Perform PCA without
> > > previously subtracting the mean.
> > >
> > > I am working with dimension reduction of images.From my experiments I am
> > > finding that reconstructions of images are much more accurate when I don't
> > > subtract the mean from the covariance when performing PCA.
> > >
> > > Can anyone help explain this?
> >
> > The reason why your analysis works better without the mean subtracted,
> > is
> > probably that images are non-negative in the first place.
> >
> > As for why the mean is subtracted in general PCA, consider two
> > orthogonal sinusoidals s1 and s2 defined as, say,
> >
> > s1 = [sqrt(3)/2 1/2]'
> > s2 = [-1/2 sqrt(3)/2]'
> >
> > A signal that comprises these sinusoidals will come out fine with
> > respect
> > to the eigenvectors of the covariance matrix, that is hermitian an thus
> >
> > have orthogonal eigenvectors.
> >
> > Now, if you add a non-zero mean m to these vectors, the vectors
> > v1 = m + s1 and v2 = m + s2 are no longer orthigonal. And so
> > the relation between the signal components and the eigenvectors
> > of the covariance matrix is no longer "easy" to deal with.
> >
> > But again, these considerations work for the parameter estimation
> > problem of sinusoidsals. They need not apply to other types
> > of signals, like images.
>
>
> This is an interesting point of view. But one can still
> find an orthogonal basis for the vectors [v1,v2]. It simply
> won't be the original trig functions. Its still just as
> valid, and will still reconstruct the data as well.

Sure. This si the Karhunen-Love Transform, if I am not mistaken.
My point is merely that one usally imposes some significance
on the eigenvecors, that holds in the zero-mean case.

This significance, which usually is the basis for whatever
elaborate analysis one is up to, is then lost in the case
of a non-zero mean.

> In fact, since the eigenvalues for the original set will be
> equal to each other, even the original trig functions may
> not be recovered, since eigenvectors corresponding to
> multiple eigenvalues are not unique.

There are several ways of getting from a set of eigenvectors
to trig functions. MUSIC uses the NULL space of the covariance
matrix of the noise-free signal, and searches for the sines
that are orthogonal to the null space.

The Kumaresan-Tufts Forward-Backward Linear prediction scheme
sets up a set of equatons from the eigenvectors of the signal
space of the covariance matrix, and solves for the frequency
terms.

But all these methods provide ambiguous results, due to
cos(-x) = cos(x), so you basically need restrictions on
the solution to get a unique answer. In some applications
it might be useful to convert the data from a real-valued
representation to a complex-valued representation to avoid
the ambiguity due to Euler's equations,

2cos(x) = exp(jx) + exp(-jx)
j2sin(x) = exp(jx) - exp(-jx)

Rune

Rune

.

Relevant Pages

  • Re: Covariance Mean subtraction in Principal Component Analysis
    ... >> covariance matrix when computing PCA. ... >> previously subtracting the mean. ... > to the eigenvectors of the covariance matrix, ...
    (comp.soft-sys.matlab)
  • Re: Covariance Mean subtraction in PCA
    ... > the Mean from the covariance matrix when computing PCA. ... > valid to Perform PCA without previously subtracting the mean. ... where U is the p x r matrix of eigenvectors ...
    (sci.stat.math)
  • Re: Covariance Mean subtraction in Principal Component Analysis
    ... > I was just wondering what is the advantage of subtracting the Mean from the covariance matrix when computing PCA. ... Is it valid to Perform PCA without previously subtracting the mean. ... to the eigenvectors of the covariance matrix, ...
    (comp.soft-sys.matlab)
  • Re: The use of PCA on a 64x64 Matrix
    ... > value of the pixels. ... and finding the mean for each column vector. ... the covariance matrix is 4096 x 4096. ... I then calculated the eigenvectors ...
    (comp.graphics.algorithms)
  • Re: nnz in convariance of a sparse matrix
    ... >> to find the eigenvectors of the covariance matrix, ... I am trying to find eigenvectors of the covariance matrix. ... Note that the multiplication is fast because I-A is a sparse matrix. ... multiplication is the power method, ...
    (sci.math.num-analysis)
Loading