Re: sum of a large covariance matrix

In article <ef3d34d.-1@xxxxxxxxxxxxxxxxxxxxxxx>, danielle
<danielleblanc2004@xxxxxxxxxxx> wrote:

hi all,

I have a 155x12000 matrix. I am unable to compute the covariance
matrix since i am running out of memory. is their a neat trick to
compute the sum of the covariance matrix (I don't need to store it.
simply i need the sum of the variance-covariance matrix) i hope
someone could help.
D
----------------------
Do you have 12000 variables and 155 observations, or is it the other way
around? The matlab function, 'cov', assumes that each row is an
observation, and each column a variable, so 'cov' of a 155 x 12000 matrix
would yield a 12000 x 12000 covariance matrix, one for each possible pair
of variables. Is that your intent? If so, that would account nicely for
your memory problem. If you mean it the other way around, you should take
the transpose of your matrix before applying 'cov' to it and doing the
summation.

In either case, there is a shortcut you can use in your computations if
all you want is the sum of all the covariance values. First, compute the
vector t, where, for each observation, the corresponding element of t is
the sum of all the variables. If N is the number of observations, t will
have N elements. If X is your matrix, this will be either t = sum(X,1) if
the columns of X are observations or t = sum(X,2) if the rows of X are the
observations. Then,

s = 1/(N-1)*(sum(t.^2)-1/N*sum(t)^2);

will give you the desired sum of all covariances without your ever having
to generate the covariances themselves, and you should have no memory
problems with this method. Also this is probably faster than having to
compute each covariance value and then add them.

Roger Stafford
.

Relevant Pages

  • Re: avoiding for loops
    ... =A0Assuming you meant that b is 40000x9 and bis correc= ... What covariance is it you estimate? ... But as Itold this takes too long time and I have memory problem I can not allocate memory for C.The virtual memory that I use is 40 GB. ...
    (comp.soft-sys.matlab)
  • Re: avoiding for loops
    ... I'm assuming you're going to operate on some subset of the pixels so that memory and run time are not concerns. ... % Pick a pair of pixels and calculate the covariance. ... % Show how to extract the covariance of pixelsover all images. ...
    (comp.soft-sys.matlab)
  • sum of a large covariance matrix
    ... matrix since i am running out of memory. ... is their a neat trick to ... compute the sum of the covariance matrix (I don't need to store it. ...
    (comp.soft-sys.matlab)
  • Re: Symbol for Covariance
    ... > Correlation Coefficient has rho. ... > Is there a symbol for Covariance? ... Cov is quite common. ...
    (sci.math.symbolic)
  • Re: modifying mahalanobis distance
    ... Hi, I am not sure if you are still reading this thread, I now understand why decreasing the error C_i on a data point y_i brings the ML solution of the mean closer to it. ... a better covariance K_seems to be harder to derive because the covariance of the cluster is inside the inversion of the sum of two matrices. ...
    (sci.math)