Re: compare 2 odds ratios

Ray Koopman wrote:
Marc Schwartz wrote:
[...] I
spent some time doing a Google and Medline search, looking for pointers
to studies where a gold standard was being compared to two alternative
tests, where the primary efficacy measure is dichotomous.

What appears to be "typical" in this situation, is the use of Cochran's
Q Test, which is a generalization of the McNemar test to >2 strata. This
test provides insight into whether or not there are differences across
the multiple strata relative to marginal homogeneity.

Cochran's Q Test can then be followed by multiple pairwise comparisons
performed with the standard McNemar test, giving due consideration to
multiple comparisons correction approaches. Thus, in this case,
performing GS vs T1, GS vs T2 and T1 vs T2.
[...]

If I understand the data layout correctly -- n Subjects x k+1 Tests,
scored 0/1, with one of the Tests being "correct" -- Cochran's test
misses the point. The null hypothesis it tests is that the k+1 Tests
have the same discovery rate, which is almost irrelevant. More to the
point would be to subtract each subject's "correct" Test score from
his k other Test scores and then do a Cochran test on the squared
values in the resulting n x k matrix. This would test the null hypo-
thesis that the k Tests have the same error rate. Pairwise comparisons
would be McNemar tests on the k(k-1)/2 pairs, with some correction for
multiplicity.

Ray,

The default null hypothesis tested by Cochran is (in this case):

H0: Pgs = Pt1 = Pt2

Where 'P' is the proportion of responses in one of the two dichotomous
response categories in the test (positive or negative), across the three
diagnostic tests (gs, t1 and t2) in use here.

Cochran is essentially providing an omnibus test of the equality of the
proportions of a dichotomous response in the 3 groups for dependent
measures.

The alternate hypothesis (HA: not H0) is that at least two of the groups
differ.

To your point, there is no impact on the resultant test relative to the proportion of the mis-classification of outcomes. In fact, in Sheskin on page 874, this is essentially referenced in Section VII in the first issue raised relative to using the test. In that section he indicates that one can remove all of the rows where the outcomes were the same, leaving only the discordant rows (akin to McNemar's) and not affect the test (since it would be the discordant rows that could affect the column proportion differences). He also notes that it is possible to obtain a significant test, even if the vast majority of rows are concordant. Thus, the nature of the test needs to be considered in that light.

I took some time today to code the Cochran's Q Test in R to play around with this process, running the test on 3 groups with varying levels of concordant and discordant rows. The formula is based upon Equation 26.1 in Sheskin on page 869. FWIW, here is the code in case anyone wants to use it:

CQ <- function(mat)
{
k <- ncol(mat)

C <- sum(colSums(mat) ^ 2)
R <- sum(rowSums(mat) ^ 2)
T <- sum(rowSums(mat))

num <- (k - 1) * ((k * C) - (T ^ 2))
den <- (k * T) - R

Q <- num / den

df <- k - 1
names(df) <- "df"
names(Q) <- "Cochran's Q"

p.val <- pchisq(Q, df, lower = FALSE)

QVAL <- list(statistic = Q, parameter = df, p.value = p.val,
method = "Cochran's Q Test for Dependent Samples",
data.name = deparse(substitute(mat)))
class(QVAL) <- "htest"
return(QVAL)
}


The argument 'mat' is simply a n x k matrix of the dichotomous outcomes, each row representing a subject, each column the outcome (0/1) for a given test/condition.

So for example, using the following matrix, with 18 subjects and three test conditions:

> NoMatch
GS T1 T2
[1,] 0 0 1
[2,] 0 0 1
[3,] 0 0 1
[4,] 0 0 1
[5,] 0 0 1
[6,] 0 0 1
[7,] 0 1 0
[8,] 0 1 0
[9,] 0 1 0
[10,] 0 1 0
[11,] 0 1 0
[12,] 0 1 0
[13,] 1 0 0
[14,] 1 0 0
[15,] 1 0 0
[16,] 1 0 0
[17,] 1 0 0
[18,] 1 0 0

We get the following result:

> CQ(NoMatch)

Cochran's Q Test for Dependent Samples

data: NoMatch
Cochran's Q = 0, df = 2, p-value = 1


Note that despite there being no concordant rows in the matrix, the test is not significant since the proportions of outcomes across the three groups are equal.

The McNemar's test is of course looking in a more focused fashion at the
discordant pairs between two groups. Along with appropriate measures of
sensitivity and specificity one can then make specific pairwise
comparisons to ascertain any differences across the tests.

HTH,

Marc
.

Relevant Pages

  • Re: What is your opinion about this ruling
    ... TD is adjusting the score and there are multiple possible outcomes, ... basis of "most favorable result that was likely" for EW (here, ...
    (rec.games.bridge)
  • Re: Which Singularity Dont You Want?
    ... then multiple possible outcomes are possible where ... In other situations (where randomness does not ... He said it entails unpredictibility, ...
    (rec.arts.sf.written)
  • Re: How would you defend yourself?
    ... multiple victims. ... # The proportion of homicides involving multiple offenders rose ...
    (talk.politics.guns)