10 Apr Inter-Rater Agreement Versus Inter-Rater Reliability
To assess the advisors` agreement, we first calculated two reliable variation indices (RPI), one based on the test reliability of the ELAN manual, the second taking CCI into account for our study population. Note that while the two reliability indicators can be used to calculate the ROI, they are not equivalent in terms of accuracy and rigor. Test-test correlations represent a very accurate estimate of the reliability of the instrument (compared to a stable construction over time), the reliability of the Interrater reflects rather the accuracy of the evaluation process. The share of the (reliable) agreement was assessed on the basis of the two estimates of honourability to show the impact of the choice of the insurance measure on the evaluation and interpretation of the agreement. In addition to the absolute proportion of compliance, information on the magnitude of the (reliable) differences and on a possible systematic orientation of the differences is also relevant to the full evaluation of the agreement of the raters. Thus, this report takes into account three aspects of agreement: the percentages of ratings that differ reliably, if any, to what extent they differ, and the direction of the difference (i.e. a systematic tendency of the two groups of advisors to react to the other). In the analyses presented here, we also refer to the size of the differences based on factors that may influence the likelihood of divergent assessments in our sample: the sex of the assessed child, the bilingual family environment, and the subgroup of raters. For rxx, we used two different reliability dimensions: (1) the RICC obtained in our study population and (2) the test test reliability (Bockmann and 0ese-Himmel, 2006), a value that comes from a larger and representative population and rather reflects the characteristics of the ELAN and not our sample.
The use of external sources of reliability indicators, as used in the second RCI calculation, was.B recommended by Maassen (2004) and can be considered the most conservative means of estimating ROI. Dijkstra PU, by Bont LG, van der Weele LT, Boering G. Common measures of mobility: reliability of a standardized method. Mr. Cranio. 1994;12:52–7. Only using the retest test reliability indicated in the ELAN manual was there a significant number of different evaluation pairs (30 out of 53 or 56.6%). The extent of these differences was descriptively assessed using a dispersal diagram (see Figure 3) and a Bland-Altman plot (also known as the Tukey Average Difference Chart, see Figure 4).
First, we presented the assessment of each child in a dispersal diagram and illustrated the two areas of agreement: 43.4% of the evaluations, which differ by less than three T-points and can therefore be considered consistent in the more conservative RCI estimate, all 100% of the evaluations are within 11 points and therefore within the limits of the agreement based on a reliability estimate. , which was determined in this study. Information about the study was e-mailed to the 250 employees of a primary care rehabilitation company in Stockholm, Sweden.