Collaborative Research: Comparing Single - vs. Double-Blind Review of Scientific Abstracts for Accuracy and Bias
To disseminate and share scientific results-whether it be via a print publication or a talk or poster at a scientific conference-the scientific community relies on an impartial and fair assessment of research quality. However, there is a genuine concern that during this peer review process, factors like gender, race, and prestige of the author or their institution can have an unwanted impact on the assessment. For this reason there are often calls for scientific communities to move from a single blind review process (where the identities of the reviewers are withheld but not the authors) to a double-blind review process (where the identities of both the reviewers and the authors are withheld). Yet, there are strong arguments for the use of single-blind review, including that it is hard to truly conceal the identity of the authors from the reviewers. Moreover, authors' identities may serve as a valid cue for the quality of the work. Thus, withholding identifying information may be unsuccessful and even impede the review process.
Currently, the Society for Judgment and Decision Making (SJDM) has been considering these arguments in deciding whether to maintain the current single blind review process of selecting talks for its annual conference or to move to a double blind system.
This project aims to take advantage of this special opportunity to run a large field experiment comparing a single-blind review process to a double-blind system in selecting talks for the next annual conference. This project will compare a double-blind and a single-blind procedure by having each submission evaluated under each procedure. This procedure will allow a direct comparison of the two procedures across all the papers in terms of accuracy and bias and at the same time help characterize the judgment process of reviewers in a high stakes selection process.