Connections with classical statistical methods

If you’ve read a statistics textbook, you might think that the methods described above are very different than classical hypothesis testing methodology. In fact, the main difference is in the way the distribution of the test statistic under \(H_0\) is obtained.

Traditional methods in statistics, use mathematical derivations (instead of simulations) to obtain the distribution of the test statistic. For example, distributions for statistics based on sample means can be approximated by a a normal distribution because of the central limit theorem. This can lead to great computational savings - it is sufficient to calculate means and standard deviations to determine an approximation to the distribution of the test statistic - but it diminishes the flexibility in the choice of test statistics.

We end this chapter with a short introduction to type 1 and 2 errors that are an important concept in classical hypothesis testing literature.

Type 1 and type 2 errors

In some studies, a decision needs to be made about which hypothesis to trust, \(H_0\) or \(H_A\). A decision means that one of the hypotheses is selected, and is usually done by selecting a cut-off value \(\alpha\), called significance level: a commonly used significance level is \(\alpha=0.05\). If the p-value is smaller than \(\alpha\), we reject \(H_0\), otherwise we accept it. Obviously, the decision might be incorrect. The table below shows scenarios when the decision is correct, and for those when they are incorrect, it defines two types of errors.

A type 1 error is rejecting the \(H_0\) when it is true. Note that the type 1 error rate is equal to the significance level. A type 2 error is failing to reject the \(H_0\) when it is false.

Type 1 and type 2 errors are different sorts of mistakes and have different consequences. In many scenarios, \(H_0\) is the status quo (for example, a new treatment has the same effect as an old treatment). Rejecting \(H_0\) possibly means that the new treatment should be implemented, but a type 1 error introduces a false conclusion and can lead to a tremendous waste of resources before further studies invalidate the original finding. A type 2 error — failing to recognize a medical breakthrough — represents a missed opportunity.