Hi Björn,

That’s not really the same point. Of course one can aggregate statistical evidence and set a meta-analytic criterion of .5%; however, I am arguing that “discovery” of an effect is a broad scientific issue, not a narrow statistical one. It requires a constellation of results, some positive, some negative — or a parametric manipulation — to assess the limits of the effect and its robustness. Only when all this work has been done can one say that an effect has been “discovered” (otherwise you don’t really know what the effect is, even if you believed it — is it an effect of this room? These stimuli? That experimenter? For these participants? Maybe an interaction between these?).

There is not really a compatibility between RSS’s “For fields where the threshold for defining statistical significance for new discoveries is P < 0.05, we propose a change to P < 0.005…” and my view, which is “discovery of an effect is not subject to a statistical criterion, because it requires assessing a constellation of results within a theoretical context”.