Discussion themes 2026

Overview

At the October 2026 meeting of the Comparative Judgement Research Consortium, participants took part in a Grand Challenges Activity led by Marie-Josée Bisson. It followed on from the Grand Challenges 2026 research agenda.

Participants split into four groups and each was given a small set of research questions and discussed:

• Which question interests you the most?
• What do you think is the next step to answer it?
• What evidence would you need to answer it?
• How would the answer to this question facilitate or advance your research?

Each participant then submitted their answers through an online form. The key themes below summarise those responses.

Key themes

1. Usability beats novelty. Requests favour a regression-style toolbox, fast algorithms, and CJ engines that support new designs; NMM was flagged for not supporting the proposed pairing strategies.

2. Multidimensionality and complexity. This was the strongest pull, framed as a tension between realism and interpretability, and it raises the question of when item complexity makes CJ unviable.

3. Validity, not just reliability. There is a clear instinct to benchmark CJ against something external: absolute judgements, control groups in schools, disciplinary theory, model-fit diagnostics, and bias detection.

4. Judges as the main source of uncertainty. Heterogeneity, bias, and expertise versus crowdsourcing arise repeatedly. Prompts appear only obliquely, seen as assuming participant expertise while also restricting how participants think.

5. Capability and collaboration gaps. Several next steps are social rather than technical: read the literature, find a collaborator, talk to practitioners. The gap between applied statistics and practice is itself a barrier.

6. A notable appetite for qualitative evidence. Think-alouds, judge comments, student reactions to feedback, and perceptions of complexity are all in demand. This is unusual in a predominantly psychometric field, and a good hook for collaboration.