Getting Away With Murder
Murder is a serious, but alarmingly prevalent phenomenon in academic science. Every minute, someone somewhere kills another person’s idea. After years of labor, a researcher puts together a paper detailing a question, data gathered and the interpretation of the results. Each paper is a partially correct story presenting an idea and supporting evidence. Now the paper is handed off to peer reviewers, who will decide the fate of the paper (and the scientists involved). These reviewers are stressed, busy, get no credit for their effort and have intrinsic biases, whether they know it or not.
A relevant and famous example of irrational judgements is the finding that parole application reviews for convicts are more frequently approved if the judge reviews the application after lunch, rather than before lunch. Like a helpless prisoner, an academic scientist can only hope their peer reviewer(s) had lunch before reviewing their paper. In short, peer review is far less rational than we think and it may help to know some of the issues.
Daniel Kahneman’s incredible book, “Thinking, Fast and Slow”, systematically summarizes decades of psychology research that shows most decision-making is highly irrational and the experience of rationality is frequently an illusion. The brain will rationalize almost any outcome. Thus, even the most data driven members of society are irrational most of the time. One of my favorite examples is the finding that none of the best Wall Street stock traders do better than chance – despite deep training, oceans of data and state-of-the-art algorithms. Nonetheless, they are hailed as experts and draw massive salaries for services that a coin toss could achieve. The implications of this research for expert peer review are substantial. Kahneman doesn’t touch on it much, but I lay out some warnings derived from the material in his book on COGNITIVE EASING, which is a major factor that shapes favorable choices:
One experiences cognitive easing when things are “perceived” as going well, because incoming information is familiar and fits with the individual’s existing knowledge of the world (See Figure). Thus, there is no need to mobilize effort to form a new understanding and no changes exist that are potentially indicative of a threat. In contrast, cognitive strain arises when one is challenged with unfamiliar information and must strain to understand the new information and fit it into their current view. Cognitive ease leads to a positive feeling and cognitive strain leads to a negative feeling and dislike. However, the experience of cognitive ease is not rational – it is a subconscious experience.
Research shows that favorable decisions are frequently based on simple things like preconceived expectations, font or language structures that promote cognitive easing and feelings of familiarity. This is a big problem when the decisions for rejection or revision should be based on the data and conclusions, and it is horrible for people trying to publish new ideas…
CONSIDER THESE POINTS:
- Cognitive easing and novelty: Truly novel directions and ideas that break from the current fashion of the field will be aversive to a reviewer only because they are unfamiliar and cause cognitive strain. In contrast, a new study that presents results that fit with the reviewer’s expectations and existing knowledge will be viewed more favorably. This experience is not rational – it is a feeling that manifests immediately and unconsciously and will increase acceptance of familiar lines of investigation, not novel ones.
- Cognitive easing and asking for more experiments: Peer review sometimes involves less of an evaluation of the evidence for the conclusions drawn and more of a laundry list of the experiments the reviewer feels would be make the paper more interesting/believable to them. How frequently are the experiments actually suggested by the reviewer because they are seeking a feeling of cognitive ease and the suggested experiments will provide it for them? This is different from requesting experiments that relate to the actual conclusions drawn by the authors.
- Cognitive easing and misreading: If the peer reviewer is stressed and struggling to understand your work, they will experience cognitive strain and be more likely reject the study independent of the data or conclusions. Frequently, they may extract conclusions that you never made or misread because they are looking for patterns that feel familiar. Their decision will ultimately be made on some sort of gut feeling rather than a rational evaluation of the content.
Rationality Is Frequently An Illusion: So what to do?
Kahneman shows that these cognitive biases are never going away and are deep in our biology. Amazingly, even if you teach people all the issues and solutions, they will convince themselves “this time is different”. However, you don’t want to be the goof that kills a big thing (or career)? (eg. initial grants to use CRISPR for mammalian genome editing were rejected). So, what can you do to fight cognitive ease?
AUTHORS:
- Accept that if you are really working on novel ideas, your colleagues won’t like them until they become familiar.
- Accept that Nature, Cell and Science are often publishing important papers that fit with the momentum of the field.
- Be persistent and don’t give up on your novel ideas! Present them widely so they become familiar to others.
- Present your work as clearly as possible and ground the novel ideas in familiar concepts.
REVIEWERS:
- Commit to signing the review with your name. If you can’t do this, it is probably because you don’t have the time to do a good job. Take responsibility for your work.
- Rationalize your arguments. This is your best defense against cognitive easing and straining biases. Generic statements that the study isn’t novel/important/high impact/rigorous are not constructive and probably the result of cognitive strain. What is the evidence for your claim(s)? Can you cite papers that directly undermine the novelty/impact? Can you defend your case with clear arguments based in fact, like a lawyer? If so, you are less likely to have been a victim of cognitive bias and the authors can consider your points carefully.
- Be constructive. Help the authors carve out a useful “knowledge product” (aka. useable knowledge). Often the language and cognitive strain is actually at the heart of your negative reaction. Realize that language is easy to change and may be causing your aversion. If their claim is too strong, what can they claim in your view? If no substantive claim is supported, then reject with a clear rational. Every paper gets better with peer review. No one wants to publish a shitty paper and they are depending on you to think of things that they didn’t – and save their bacon. It is OK to kill someone’s idea….just make sure it is clear why you did it and how to move forward.
Christopher Gregg, PhD