Re: permutation p value vs. normal p values

On 16 Aug 2005 12:39:11 -0700, arrayprofile@xxxxxxxxx wrote:

> We have follow-up information (survival time and censoring) for each
> patient, so we can use Cox regression to evaluate the correlation of
> each gene's expression levels with patient's survival. The sample size
> is indeed small, but that's what we have right now. The actual number
> of genes is 15338, so by chance alone, I would expect 15338*0.05=767
> genes with a p value less than 0.05. With permutated p values, I had
> 763 genes with a p value less than 0.05, so they are very very close.

Getting 5% rejections (at 5%) by permutation usually says
that the test is okay -- unlike the no-power tests described
below -- and that the data are random.

> Does this mean the permutations I did really generated a good
> distribution under the null hypothesis? What puzzles me is that the p
> values from Cox using normal distribution only produced 2 genes with a
> p value less than 0.05. What would be the reason for this difference?

Well, I'm sure that getting only 2 was not an accident.
Or, the only accident was why *2* did show up as significant,
instead of none.

As I suggested the first time, I now conclude (if I didn't
conclude it then) that you must have a test that has
NO POWER -- like asking if 3 coin flips are significantly
all Head or all Tails, the answer, 15000 times, is NO.


My first guess is that you are mis-applying the Cox Regression.

I see several possible difficulties, given what you have
said and the little that I know about Microarrays --

A Cox regression tests 2 or more groups, assuming equal
proportions in Death (relapse) at each time period.

1) If you typically have 17 "groups" for expression levels,
that gives zero power for finding differences. Is expression
measured as Yes/No, or as a continuous score?

2) If you don't have multiple deaths at any period, you also
have zero power. I don't think that Cox Regression has
any cure for that, since the death rates at each period
are independent. Use no more than 2 Times?

3) If you don't have very many deaths, your power is very
low, since it is limited by "Deaths" as much as it is limited
by total N of 17. For Cox, it should be limited also by
Deaths per period.

I suspect that your permutations are irrelevant to whatever
you are accidentally, actually testing with Cox.

If (a) your Gene Expression is measured by a continuous score,
and (b) censoring is all at a fixed time, and (c) half the sample
Died, then you might have better luck by testing Expression with
a t-test on Died/Survived.

--
Rich Ulrich, wpilib@xxxxxxxx
http://www.pitt.edu/~wpilib/index.html
.