Re: intraclass correlation

On 12 Apr 2006 09:34:58 -0700, "Abu" <amsaslist@xxxxxxxxx> wrote:

SAS macro %INTRACC computes 6 different intraclass correlation
coefficients discussed in Shrout and Fleiss "Intraclass correlations:
uses in assessing rater reliability," Psychological Bulletin, 1979, 86,
420-428.

When you use the sas macro intracc, you get the intraclass correlation
coefficients for the ratings of n raters and also for the average
ratings of n raters. The second one obviously averaging out the rater
variability and results in a much higher value of icc. I just can't
find a justification for doing that. Do you know of any reference that
discusses the advantages/disadvantages of each of these methods? When
do I use the average method as opposed to the individual ratings?

A study may collect data with multiple raters, as pilot
data or on a subset of cases, solely for the purpose
of computing an ICC. In that case, the single-rater ICC
is appropriate.

Sometimes a study intends to use multiple raters at
every data collection opportunity in order to boost the
reliability of the measures. In that case, the multi-rater
ICC from the pilot data would be useful for the statistical
power analysis -- the single-rater ICC would underestimate
the reliability. Even when the multi-rater ICC has that use,
I still like to see the single-rater ICC included in the study
results. Actually, I like to see average Pearson r and how
r varies between raters, since that is the statistic that
readers are most likely to be familiar with. (Plus, it is nice
to have paired t-tests on means, or whatever, as further
assurance that the mean levels are consistent, if that matters
for the design.)

If Shrout and Fleiss don't have comments on when to use
which ICC, I would expect them to cite references that do.

--
Rich Ulrich, wpilib@xxxxxxxx
http://www.pitt.edu/~wpilib/index.html
.

Relevant Pages

  • Re: intraclass correlation coefficient
    ... ICC value for single rater? ... or reliability coefficient ... defending a scale at the time of publication? ... raters, when single-rater scores are going to be used. ...
    (sci.stat.math)
  • Re: kappa & ICC questions
    ... Can the data from more than one pairs of raters in an inter-rater ... reliability study ... There exists a creature called a multi-rater kappa. ... rated by each rater in the test-retest reliability study ranges from 9 ...
    (sci.stat.edu)
  • Re: interrater reliability
    ... >> I am wondering how to evaluate interrater reliability ... My study purpose is to evaluate the interrater ... >poor if the Subjects are nearly identical ... How often are the raters ...
    (sci.stat.math)
  • Re: test re test reliability
    ... intra class correlation is being advocated as ... > a more accurate expression of reliability over time as it also ... raters is good for planning for multiple raters. ...
    (sci.stat.edu)
  • Inter-rater reliability question
    ... If I have three columns of data representing three different raters, ... conduct an "inter-rater" reliability? ...
    (microsoft.public.excel.worksheet.functions)