Re: Agreement for one subject with gold standard, ordinal data

Thom wrote:
> I wonder whether standard correlation coefficients do the trick?
>
> With this kind of exercise isn't the most critical information order
> dependent? In other words an inversion at the beginning might be a more
> critical error than one at the end?
>
> Would some kind of weighted scoring be desirable?
One way would be to weight the squared differences between the gold
ranks and a subject's ranks by the reciprocal of the gold ranks;
i.e., sum (x - y)^2/x, where x is gold. But note that this could not
be used to measure the difference between two subjects' rankings.

Another way would be to convert all the ranks to (reverse) Savage
scores (n -> 1/n, n-1 -> 1/n + 1/(n-1), ...; or just use the logs
of the ranks, which are almost linear with Savage scores and may be
a little easier to understand) and then use the unweighted sum of
squared differences (or, equivalently, the Pearson r). This would
allow any two rankings to be compared.

.