The international network for comparative judgement research
Comparative Judgement Research Consortium
An international consortium of researchers and practitioners who use, apply and develop comparative judgement methods.
The Comparative Judgement Research Consortium (CJRC) brings together researchers and practitioners from statistics, education, psychology, linguistics, political science, computer science and beyond, united by a shared method: comparative judgement. We run regular meetings across the UK and Europe, maintain the largest open collection of comparative judgement datasets, and set the research agenda for the field. The consortium is convened by Ian Jones (Loughborough University), Marie-Josée Bisson (Loughborough University) and Rowland Seymour (University of Birmingham).
What we do
Meetings
Full-day meetings several times a year, hosted by member institutions, with talks from across the disciplines and structured discussion. The next is at Loughborough University on 13 October 2026.
Grand Challenges
In May 2026, 25 experts from statistics, education, psychology, politics and computer science co-created a research agenda for comparative judgement: 36 open questions in seven themes.
Reading group
An online reading group, running since 2024, discussing papers from Thurstone (1927) to the latest work on comparative judgement and large language models.
Resources
A curated data repository, a bibliography, a guide to software for collecting and analysing comparisons, and guidance for researchers new to the method.
What is comparative judgement?
Comparative judgement is a method for measuring quality, or any other latent trait, by asking judges to compare items in pairs rather than to score them one at a time. Its foundations go back to Thurstone’s (1927) law of comparative judgement, and it rests on a simple observation: people are far more consistent at deciding which of two things is better than at placing one thing on an absolute scale.
- Judges are shown two items, such as essays, proofs, images or research proposals, and choose which is better against a holistic criterion.
- Many such decisions are collected from many judges, with each item appearing in several pairs.
- A statistical model, typically Bradley–Terry or a Bayesian extension of it, combines the decisions into a measurement scale for the items and estimates its reliability.
Comparative judgement is now used well beyond its origins in educational assessment: in psychology to measure conceptual understanding, in linguistics to study language processing, in political science to measure attitudes and priorities, in statistics to model everything from urban deprivation to sports results, and increasingly to evaluate the outputs of large language models. Members of the consortium have published on all of these; our bibliography is a good place to start, and our guidance page explains how to run a first study.
Join the consortium
Membership is open to anyone working with, or interested in, comparative judgement. Joining our mailing list brings news of meetings, the reading group and calls for collaboration.
Alternatively, email LISTSERV@JISCMAIL.AC.UK with the message body SUBSCRIBE COMPARATIVE-JUDGEMENT FirstName LastName.
Acknowledgements
Our work is supported by the Engineering and Physical Sciences Research Council through a Mathematical Sciences Discipline Hopping Grant [UKRI2389] and a UKRI Future Leaders Fellowship [MR/X034992/1]. The consortium was established with a National Centre for Research Methods Special Interest Group grant, and our meetings have been hosted and funded by the London Mathematical Society, Bath Spa University, De Montfort University, the University of Birmingham, Loughborough University and the Université de Liège.
