by Mathew Barlow, UML Professor of Climate Science
The basic unit of scientific evidence is a peer-reviewed publication in a reputable scientific journal. More on exactly what that means in a moment, but I want to first emphasize that a peer-reviewed journal publication (or just “paper,” for short) is the beginning of the story, not the end.
The gold standard for scientific evidence is multiple papers, by multiple different groups, using multiple different methods. That is the point at which the scientific community starts to feel there is strong evidence for a particular result, or idea, or interpretation.
Why? Let’s return to the idea of peer-review and journal reputation.
Peer review is the process where other scientists with relevant expertise (the “peers”) read over a paper before it is published and assess whether it is generally reasonable, usually through an iterative process of questions from those scientists and responses and edits from the authors (the “review”). This process is supposed to establish that the paper generally seems reasonable and follows accepted scientific practice but it does not, and is not intended to, guarantee that the results are right, even when the process works correctly (and it doesn’t always). It is basically a sanity check and minimum quality control, although it does often considerably improve the papers (and frequently frustrate the authors, reviewers, and editor). A crucial and intentional part of the scientific process is the testing of a result by other groups, using other methods. Science is a messy, human process, where people sometimes have agendas, every method has its pros and cons, the data from a single experiment are rarely unequivocal, and there is heavy pressure from institutions and funders for frequent publication and novel results. When different groups get the same answer with different approaches, those concerns are directly addressed.
Keep that in mind when you read about a new, high-profile result – again, that’s the beginning of the story, not the end.
Finally, a few words about whether a journal is “reputable.” This used to be less of a concern but there is a lot of money to be made in publishing, and there are now a vast number of journals, some of which have reasonable sounding names, a reasonable-looking website, and claim to do real peer-review – but don’t. This really requires a detailed discussion but, for brevity, I’ll just note that this can be a concern when evaluating papers, and is another issue that can be alleviated by independent groups reproducing the same result (in different journals). Failure to fully disclose the funding source behind a paper, which is relevant to assessing intentional or unintentional bias in an analysis, can be an issue even for reputable journals, and is even harder to track.
Further reading
For more introductory material on peer review:
https://guides.lib.jjay.cuny.edu/c.php?g=288333&p=1922599
For an inside view, here are the AMS guidelines for reviewers:
Assessing journal reputation and whether a journal can be considered “predatory” is a complicated issue, muddied further by all the money at stake, including for big name academic publishers, whose business model is challenged by open access publishing – which includes both legitimate and predatory journals.
Here’s some introductory material on predatory journals: