A theory-based intervention to prevent calibration effects in serial sport performance evaluations

Objectives: Serial performance evaluations show calibration effects: Judges avoid extreme categories in
the beginning (e.g. best or worst) because they need to calibrate an internal judgment scale (Unkelbach
et al., 2012). Successful calibration is therefore important for fair and unbiased evaluations. A central
prerequisite for successful calibration is knowledge about the performance range. The present study tests
whether advance knowledge about the range (best and worst) of performances in a series reduces
calibration effects.
Design: A 2 2 2 design was developed with two between subject factors: the knowledge about the
performance range (with vs. without) and two different talent tests (specific vs. unspecific). As within
subject factor the position of the performances in the series (position 1e10 vs. 11e20) was integrated.
The combination of the between subject factors resulted in four experimental conditions.
Method: Handball coaches were randomly assigned to one of the conditions. Afterwards twenty performances
were evaluated in a randomized order by the coaches.
Results: Without knowledge about the range, they showed the expected avoidance of extreme categories
in the beginning independent of the presented talent test. However, observing the best and worst
performance in advance prevented the biases. Range-presentation is therefore a viable theory-based
intervention to improve fairness in serial judgments.