Re: Are graded clinical signs more reliable than dichotomized?

John,

What if the real item is a continous variable (1-10), but forced into
two grades (0-1)? There does not exist any clear definition for the
"true" cutoffs. Here an example where two examiners apply different
cutoffs (with some error). What happens if the examiners instead can
use a 4 graded scale? Does it increase the agreement? I am not a
statistician but need your help to describe what happens when you grade
(continous) clinical signs into more than two grades?


x y1 y2 y3 y4
------------
1 0 0 1 1
2 0 0 1 1
3 1 0 1 2
4 0 0 2 2
5 1 0 2 3
6 1 2 3 3
7 1 0 3 4
8 1 1 4 3
9 1 1 4 4
10 1 1 4 4

Regards

Roland Andersson


John Uebersax wrote:
Frank E Harrell Jr wrote:
Generally speaking, having more categories improves every aspect of
prediction and decision making,

Here's a point I don't think has yet been raised. What if the trait is
a true dichotomy, or nearly so?

Let x denote a latent (not directly observed) trait with two levels,
and let it be measured by two fallible measures, y1 and y2.

The fallible measurement introduces error:

y1(i) = x(i) + e1(i)
y2(i) = x(i) + e2(i)

where e1 and e2 denote measurement error and (i) denotes some case i.
We assume e1 and e2 are uncorrelated.

Example:

Let the two latent trait levels be 3 and 8 on a 10-point scale. Let
measurement error variance for y1 and y2 be equal.

Hypothetical data for several cases might look like this:

x y1 y2
------------
3 1 3
3 2 2
3 3 4
3 4 5
3 5 1
8 6 8
8 7 9
8 8 10
8 9 7
8 10 8

Ideally one would choose numbers above such that y1 and y2 are
completely uncorrelated within each level of x, as that is the
implication of the measurement model. But the point is that r(y1, y2)
is less than 1, and these numbers suffice to show that.

Now suppose we optimally discretize y1 and y2. Then we have:

x y1 y2
------------
3 3 3
3 3 3
3 3 3
3 3 3
3 3 3
8 8 8
8 8 8
8 8 8
8 8 8
8 8 8

and r(y1, y2) = 1. I would consider this "better agreement" obtained
by having fewer levels. The recoded variables y1 and y2 also now
correlate better with x, so that we would consider them more valid as
measures of x.

It's simpler if we use 1 and 2 instead of 3 and 8, but I leave it this
way because this seems potentially closer to the point about
considering the magnitude of disagreement.

Note that using the ICC here instead of the Pearson correlation the
conclusion is the same.

Do I make heroic assumptions? Yes. But only to potentially establish
the principle that discretization is sometimes better. If so, that
places the question in the realm of considering particular data, and
not applying a universal rule.

Unless I miss some obvious point, this example seems to demonstrate
that if one has a trait which is fundamentally discrete or strongly
multi-modal, then a rating system with fewer levels can be more
reliable and more valid.

Without elaborating, let me also suggest an analogy to digital signal
filtering with a low-pass filter. It seems easily verified that
sometimes one prefers a filtered signal with coarser resolution than a
noisier signal with higher resolution. I don't want to pursue this
analogy, I just propose it for consideration.

--
John Uebersax PhD

.

Relevant Pages
Loading