Peer review is often, incorrectly, used as a gold standard by which the legitimacy of scientific studies are measure, but peer review itself provides minimal value towards establishing the veracity of a study’s findings.

The essence of science is systematically and empirically testing hypotheses of observable, repeateable, natural phenomena.

The legitimacy of any particular scientific finding is whether the application of the methods used in the study will result in similar findings upon repetition. If a study can be replicated, the study’s findings are verified.

Peer review does not replicate studies, so it does not speak to the veracity of scientific findings; it does not effect the legitimacy of a study’s findings.

Peer review does have it uses, as a review of methodology and a post-study review of the analysis of the findings, but not as a verification of the study itself. Peer review can be useful for ascertaining whether the methodology of a study actually is measuring what it purports to measure and whether the analysis of the findings are legitimate, but the value of both of these assumes the findings are verified.

You will then notice that peer review in practice occurs at improper times for both of these. For a methodological review, methodology should be fully set in place before the experiment is carried out, as any change in methodology will effect the findings and outputs. Methodological peer review should occur before an experiment begins. Analysis should occur after an experiment is verified, as analyzing false findings just leads to false analyses.  Because of this analytical peer review should only occur after findings have already been verified.

Yet peer review as practiced occurs after an experiment has been carried out, but before an experiment is verified, the worst of both worlds. It is useless for critiqing methodology, short of rejecting the study completely, as post-experiment revisions to methodology are anti-scientific. It is also useless for critiquing analyses because it is unknown if the findings being analyzed are of any actual value. It accomplishes nothing besides providing a false sense of legitimacy.

On top of this, peer review acts as a filter for what is novel, important, and, most importantly, relevant. This filter is anti-scientific. A study finding nothing useful, may not be as practically relevant as a a study finding something novel, important, and relevant, but it is as methodologically relevant. You can not filter out the studies finding “nothing” and filter in only studies finding “something”, and expect to have an accurate view of the world.

For illustration, if 10 studies are done on the same aspect of Topic X, and 9 find nothing novel, important, or relevant about that aspect of X, and 1 study finds something very novel, important, and relevant, that 1 study may pass review and be published, while the other 9 studies will not, providing a very biased view of Topic X.

As bad, this incentivizes scientists to do studies that are novel, important, and relevant, instead of studies that aren’t. Publish or perish. This creates some very obvious perverse incentives and pre-filtering effects scientific studies.

Peer review is likely, in practice, a negative on science as it creates a false sense of legitimacy which does not exist due to the lack of replication and verification. By being a gold standard, peer review gives legitimacy to a study which it has not earned. Implying a study meets the standard of being scientific, when none of its findings have actually been scientifically verified, is absurd, especially when peer review is the norm, but replication isn’t.

The entire process of scientific peer review as it currently stands should be torn down and replaced by mandatory pre-experiment methodological review and mandatory post-experiment replication. Post-replication analytical review would be good, but not necessary, for proper science. Any study that completes pre-experiment methodological review should be published, at least in summary, even if it provides nothing novel, important, or relevant, as a lack of results is just as methodologically important as “real” results.

Of course, this would be more time consuming and expensive, but this is price of doing real science, rather than pretending to do science.

Post-script: Peer review for the humanities is fine, as is, as these fields are unscientific in the first place. In an unscientific field, paper, or study, a post-study review by experts is probably useful.