To you, on the occasion of your replication crisis
“I am very sorry, Pyrophilus…”
The seventeenth century was an exciting time in Europe. A new empirical philosophy — what we call science — was emerging. Oldenburg founded the Philosophical Transactions of the Royal Society, now the oldest scientific journal still being published, to communicate peoples’ observations and ideas. People like Newton, Halley, Hooke, Boyle — names that are now immortal— were performing experiments and developing theories that would change the world.
We learn about their ideas in our science classes, filtered through centuries of additional learning, as fully-formed. We know, however, that science does not work like this: every useful regularity is underpinned by many experiments to establish its boundary conditions; every useful theory is underpinned by many such regularities. Nature does not give up its secrets easily, at least to us.
Recently, many have described science as experiencing a “replication crisis”: experiments that were believed to be trustworthy do not replicate. We have spent years trying to understand why this might be, and rightly so. Poor incentive structures for scientists? Weak theory? Variability inherent to what is being studied? Methodological errors? All of these likely play a role to some extent; the main questions involve where, to what extent, and how to reduce them.
This is discouraging to many people (myself included). It has a particularly dramatic impact on early career researchers, who try to build on previous work, only to have the experiment fail. This happened to me during my PhD, as I described on twitter:
Robert Boyle experienced the same difficulties we are experiencing now in his chemical experiments. To an outsider, this might be surprising, given how exact a science we consider chemistry: we can obtain pure materials, at exact amounts, under tightly controlled conditions. We have a theory of matter that tells us what chemicals are, and how they combine. Gold, for instance, is gold because of the number of protons in its atomic nucleus. It has predictable properties based on the “arrangement” of electrons around the nucleus. But imagine trying to learn about chemistry without knowing what a pure material is, or what sorts of things affect chemical properties? Without an atomic theory?
On reflection, one can see how difficult this would be. One might guess without knowing scientific history that it would be impossible. Indeed, it seems like you need to know about the world in order to learn anything at all from it, so it surprises me that we ever make progress. And yet, we seem to.
But progress in science is not without great discouragement. Boyle knew this well; in fact, he wrote several essays — in the style of an open letter to a young researcher — on “unsuccessful experiments”. He opens the first of these:
“I am very sorry, Pyrophilus, that to the many…difficulties which you may meet with…in…experimental philosophy, I must add one discouragement more[:]…false or unsuccessful…observations and experiments…which…disappoint your expectation, either not at all succeeding constantly, or at least varying much from what you expected.”
This is a letter from Boyle, across almost four hundred years, to us. Boyle was at the fuzzy boundary between alchemy and chemistry, having to discern which reports were reliable and which was not; which methods worked, and which didn’t; all in the context of a primitive, alchemical knowledge of matter. He describes the interesting difficulties he has faced in two broad categories:
- “[P]articular or mistaken properties of the materials employed”: that is, lack of scientific or theoretical understanding (remember, no one had a good theoretical understanding of matter at the time), and
- “[S]ome such error committed in the handling of these materials”: e.g., methodological variance.
He then distinguishes the above, which are difficult to detect, from the “more obvious ones, which proceeding barely from the unskilfulness of the tryers of the experiments, [which] may be easily enough discerned [by an expert].”
Boyle focuses on the two hard problems, outlining his experiences, admitting his own errors or misunderstandings, and making some guesses about why some of his experiences fail. His purpose is humane; he writes to Pyrophilus
“…not only evince the truth of [failures to replicate], but somewhat lessen the despondency it is apt to produce, by letting you see, that though some of your experiments should not always prove constant, you have many partners in that infelicity, who have not been discouraged by it.”
Perhaps it will somewhat “lessen the despondency” of those whose experiments seem unsuccessful now to know that Boyle was one of them. Perhaps it will even be inspiring to know that on Boyle’s many failures, and those of his contemporaries, was born a new science. What will we build as we emerge from the current sense of crisis?
Whatever it is, I hope that “failure” is normalized; as Boyle points out, it is part of the scientific process. What matters is how we handle the failures.
I have lightly edited and annotated Boyle’s first essay and placed it online for everyone to read. The original text of several of the essays in PDF format, can be found here. This is a work in progress, and I plan to release easier-to-read versions of several of Boyle’s essays soon. I would be happy to receive any corrections or comments. I am neither a seventeeth century alchemist nor a chemist.
