Why the push for replacing “power” with “precision” is misguided — [Code for all the plots in this post is available in this gist.] One of the common claims of anti-significance-testing reformers is that power analysis is flawed, and that we should be planning for study “precision” instead. I think this is wrong for several reasons that I will outline here…

“I am very sorry, Pyrophilus…” — The seventeenth century was an exciting time in Europe. A new empirical philosophy — what we call science — was emerging. Oldenburg founded the Philosophical Transactions of the Royal Society, now the oldest scientific journal still being published, to communicate peoples’ observations and ideas. People like Newton, Halley, Hooke, Boyle…

How scientists successfully reason about sampling distributions — Rink Hoekstra (@rinkhoekstra) and I are pleased to release a new pre-print, “Use of significance test logic by scientists in a novel reasoning task”. We test the ability of a large, diverse group of researchers to reason from sampling distributions, a core aspect of significance logic. Although it is common…

Ensuring our critiques of power are relevant, clear, and based on good statistics — On Twitter recently, I criticised the common practice (some might say “shorthand”) of claiming that an experiment or study is “underpowered”. This ended up being a popular tweet, which I wasn’t expecting (it is a rather arcane statistical point) but given its centrality to some of the recent issues in…

A Christmas-themed statistical game and experiment — (The experiment is over, but you can see a version of it on github) It’s nearly Christmas, but Santa has a problem: his elves have stopped working! The elves have have divided into two gangs, the Jinglies and the Sparklies, each believing that their gang can make toys faster. Santa…

Notes on Tour 1 of “Statistical Inference as Severe Testing” — Anyone who’s had any contact with statistical methods recently knows that there’s a battle being fought over the future of statistical methods. Actually, more than one; the big ones are significance testing vs confidence intervals and Bayes vs frequentism. The so-called “replication crisis” in the various sciences has provided an…

Scientists defend two researchers trying to clean up the scientific literature — [Update: the paper in question was officially retracted while this letter was being circulated; but OSU’s complaint is still outstanding. The good news is that both Markey and Elson’s Universities have declined to act on OSU’s complaint. To date, however, they have not made any public statement in support of…

Part two of a three part series — [Part one of this series is here] At the heart of the RSS paper are a number of statistical arguments. There are three, and I will address each of them in this (rather long) post. They are 1) two-sided p values around .05 are evidentially weak, 2) using a lower…

Should we “redefine” statistical significance? — [Part two of this series is here] Recently, a zillion-author manuscript came down the pipe suggesting that a change to the common practice of using p<.05 as a criterion for statistical significance. The authors include theoretical and applied statisticians and scientists, and the paper has clearly made an impression, both…