Alexander Ly posted something on the Facebook group “Psychological Methods Discussion Group” about this post, but I am not a member of that group, so I will respond here about it.

He says that I “incorrectly call[] confidence intervals the inversion of a hypothesis test,” and says that this is incorrect because the rejection region of a hypothesis test does not depend on the data. Yes, the rejection region does not not depend on the data. But the collection of null hypotheses that would not be rejected by an α-test does depend on the data. This is the sense in which the confidence interval and the hypotheses test are inversions of one another. If we consider the confidence interval, it will correspond to all null hypothesis for which p>α (for a 100(1-α)% CI). p values and CIs have a one to one relationship with one another. This is standard parlance in statistics, and it strange to take issue with it.

I believe the confusion must come from the fact that p values are often perceived as only being used/useful in a test of a single null hypothesis. But this is not the case, as I mentioned in the post.

Alexander also says that “The main problem…is that he does not give a proper defenition of bias”. This point will have to be taken up with Ward Edwards, or Jim Berger who is now the typical one to make this claim. I take it to mean simply that the null hypothesis is easier to reject than it should be, or than other values. The first argument takes care of the first definition, and the second agument takes care of the second definition.

But this cannot be used as a defense of the original argument itself; if it was not clear what “bias” meant in the first place, the onus is not anyone who claims that classical tests are baised against the null to defend the claim. If definition is not clear, then my statement that it is not a good argument is even stronger.