Re: explaining p-values for post hoc comparisons

On 28 Sep 2005 08:08:29 -0700, "Bill Howells" <whowells@xxxxxxxxx>
wrote:

> I have a running dialogue with my supervisor, a clinician, who is fond
> of finding curious things in the data and wanting p-values. I am
> fighting the good fight to promote confidence intervals and a priori
> hypothesis testing. Hooray for me! In the course of argument, I have
> made the following statement, "P-values are strictly correct only for a
> priori hypotheses", which usually produces a blank expression. It just
> isn't intuitive how the same calculation can mean different things
> based on the time of day it is done, even if the calculation later in
> the day is done after examining the data.
>
> I am at a loss to follow it up with some pithy statement that will
> convince a clinician it might be true. Things that are convincing to
> me probably don't seem convincing to a clinician. The formula,
> (1-(1-alpha))^n where n=#tests and alpha=Type I error for each test
> convinces me that Type I error for a series of tests is inflated, but
> don't think it would work on a clinician. Thoughts or advice on this
> issue? Bill H, MS, Wash U Med, St Louis

I was going to suggest, "Generate a bunch of random data."

But you already have data on hand with a lot of variables,
so I will suggest instead, "generate a bunch of artificial
dichotomies." Compare your own even IDs to the odd ones;
compare the first half to the second half (but that might
be meaningful -- think about that one); or make dichotomies
from generated random numbers.

For "correlations" instead of t-tests, you can generate 10
or 20 random 'normal' numbers, and show that there are a
bunch of random correlations with your own variables.

Moreover, the random associations, done either way, will
tend to occur in sets -- If you have sets of highly correlated
indicators among your own measures, something correlating
highly with one of them will automatically have *some*
correlation with the others.

--
Rich Ulrich, wpilib@xxxxxxxx
http://www.pitt.edu/~wpilib/index.html
.

Relevant Pages

  • Re: explaining p-values for post hoc comparisons
    ... Bill Howells wrote: ... of finding curious things in the data and wanting p-values. ... me probably don't seem convincing to a clinician. ...
    (sci.stat.consult)
  • explaining p-values for post hoc comparisons
    ... I have a running dialogue with my supervisor, a clinician, who is fond ... of finding curious things in the data and wanting p-values. ... isn't intuitive how the same calculation can mean different things ... me probably don't seem convincing to a clinician. ...
    (sci.stat.consult)