Why adopt a double-blind peer-reviewing process?
Several studies have shown the existence of (unconscious) biases in the peer reviewing process in computer science. Both positive and negative biases have been identified. For instance, in a controlled experiment at WSDM 2017 (50% of the PC single-blind, the other 50% double-blind), Tomkins et al. (2017) found that
- single-blind reviewers enter fewer bids;
- single-blind reviewers preferentially bid on papers from top universities and companies; and
- single-blind reviewers are more likely to assign a positive score to a paper of a famous author or from a top university or company.
WSDM is a computer science venue; in the absence of a controlled study in the real-time community, it is reasonable to assume that similar unconscious biases exist in our community.
Other studies have further shown biasing effects related to author gender, author ethnicity, author nationality etc. Lee et al. (2013) provide a comprehensive review of the scientific literature on bias in the peer reviewing process. Recently, Le Goues et al. (2018) discussed the effects of introducing a double-blind peer-reviewing process at ASE, OOPSLA, and PLDI, and arrived at two important conclusions: first,
while anonymization is imperfect, it is fairly effective,
the extra administrative burden [is] relatively minor and well worth the benefits.
For these reasons, the IEEE TCRTS has decided to introduce a double-blind peer-reviewing process at RTAS 2022.
Is the real-time community large enough for an effective double-blind peer-reviewing process?
How are Conflicts of Interests (CoIs) discovered?
- Authors will be asked to declare CoIs with any PC members during the submission process.
- In addition to the regular bidding phase, PC members will be asked to declare CoIs based on a list of authors and affiliations. This list will not include paper titles, abstracts, or paper IDs. This allows CoIs to be discovered that authors failed to declare or were unaware of without de-blinding any papers.
How will reviewers judge whether a contribution is “incremental”?
One frequent concern is that reviewers may have a hard time judging whether a paper’s contribution is substantial enough to merit publication at a top conference if they do not know the authors’ prior work. In the process adopted for RTAS 2022, this is not actually a problem, for two reasons:
- Author identity does not determine whether a contribution is incremental; rather, the delta relative to the state of the art does. A contribution does not become less (or more) incremental only because it extends one’s own prior work.
- Since references are not blinded, authors can (and must) discuss the state of the art objectively, and reviewers will be able to fully assess the state of the art before arriving at their decision.
How will reviewers judge whether a contribution is correct?
In some discussions, the point has been raised that it can be difficult to place much trust in the correctness of a proof without knowing the identity of the authors. This is exactly the kind of positive or negative identity-based bias that a double-blind reviewing process seeks to eliminate. Either a proof is clear and convincing, or it is not. “Proof by authority “ is not an acceptable scientific method, nor are prior mistakes sufficient justification for rejection of new results.
Similarly, some have worried that it may be difficult to weigh the validity of experiments without knowing the authors or lab behind a study. Again, a study’s merits, limitations, and weaknesses should be clearly apparent from (and only from) the submitted paper. Author identity plays no role in the assessment of empirical work.
What if some papers can be de-blinded anyway?
No double-blind peer-reviewing process is perfect. Papers by the best-known authors with distinctive styles can still remain recognizable to many. Similarly, authors working for a long time on highly specialized topics that few other authors work on will be recognizable to area experts. There are many other ways in which submissions might occasionally become de-blinded (by a subset of the reviewers), e.g., in case of a resubmission of a paper previously rejected from a single-blind conference or due to a pre-existing online preprint and an honest search for related work by a reviewer working on a similar topic. Such cases cannot always be prevented.
The important point, however, is that, to be beneficial overall, a double-blind peer-reviewing process does not have to be perfect. In particular:
- A double-blind process allows authors who believe that they might be at a disadvantage to craft a paper that cannot be attributed to anyone in particular with high likelihood.
- The fact that the process is double-blind changes reviewer attitudes and (subconsciously) reminds everyone that (suspected) author identity should play no role in the assessment.
- Double-blind submissions encourage authors to present the state of the art and prior work in a balanced way.
- Even if a paper is likely authored by a certain author or group, the fact that it plausibly could also have been authored by someone else in most cases creates enough uncertainty to deemphasize the role of author identities in the decision-making process.
What is the double-blind policy for reviewers?
Reviewers will not intentionally try to de-blind a paper (e.g., by searching for key phrases on the web). Nonetheless, as discussed above, there may still be situations where some of the reviewers feel like they have a pretty good idea about who (some of) the authors of a paper assigned to them may be. In such situations, the following rules apply.
- If PC members feel that they can de-blind a submission based on title and abstract alone (e.g., because they reviewed a prior version of the paper for a single-blind conference, or because they had a hallway conversation with the authors at a some conference prior to submission, etc.), they may still bid for the paper and should not declare a conflict of interest (at least not for this reason) because their expertise is still required.
- If PC members feel that they can de-blind a submission for any reason, they should not reveal their suspicion to anyone and should not let it influence their assessment of the work.
- In particular, if a PC member happens to have reviewed a prior version of a submission for another conference, the PC member should not reveal this information and should be careful to base her/his assessment exclusively on the present version.
What about track chairs?
The double-blind policy does not extend to track chairs (one of which is the PC Chair): as track chairs organize the peer-reviewing process, they know all author identities and affiliations (of papers submitted to their track), know all reviewers, assign papers to reviewers, and lead the final discussions. As an exception, track chairs of course will not know the reviewers for any papers that they have a conflict-of-interest with. Chair conflicts will continue to be handled as usual (by delegating such papers to another track chair or a senior, non-conflicted member of the PC).