Johnson and Omland: Why everything you know about P-values is stupid

Johnson and Omland argue for a new model of evaluating statistical tests instead of the traditional P-value tests against the null hypothesis. While we are used to tests with a null hypothesis, it is typically biologically irrelevant and its rejection is only indirect support for another alternative hypothesis. Furthermore, rejection happens at an arbitrary threshold and can be ineffective at evaluating incredibly complex and dynamic systems that biologists routinely study in nature.




And out of the wasteland of useless null hypotheses, “model selection” emerges as a saving light! Since it is able to evaluate and select a best-fit model out of a number of candidate models the researcher has proposed, it can evaluate complex situations and multiple influences. After using a simple X2 test or a parametric bootstrap procedure to fit the global model to the data, individual models can be evaluated by Least Squares to determine their effectiveness, and the Schwarz and Akaike Information Criterion can synthesize several models simultaneously into a new “best set” of available models.


Johnson and Omland highlight model selection’s successes in biological fields. In ecology, model selection is extremely useful in modeling population cycles, and abundance and survival probabilities. By modeling different survival and encounter probabilities, researchers can separate the probability of marked animal death against the probability of the marked animal surviving but not being recaptured. In addition, model selection is extremely useful for phylogenetic trees, where the order of each mutation is treated as a separate model and evaluated against one another for best-fit. Lastly, they mention that it can also be used for identifying selective pressures and adaptation in the wild.


In their second to last section, they highlight fields that should have converted to the wonders of model selection, but are stuck in the heathen ways of null hypothesis tests. In particular, they cite statistical phylogeography as being able to use current genetic distributions to recreate possible historical populations. In ecosystem science, model selection would be able to evaluate complex trophic systems and multiple food-web models.


Johnson and Omland do acknowledge a few pitfalls about using model selection, but give very little time addressing these issues (apparently leaving it to Anderson and Burnham). First, they note that the conclusions can only be as the multiple models the researcher develops before testing the data. However, I see the eagerness to jump onto the model-selection bandwagon can start from poor initial hypotheses and relying solely on post-hoc conclusions. Secondly, they state that the predictions must actually be biologically plausible and not “merely” statistical significant. If these are not plausible but are “statistically significant” this should indicate either a flaw with this modeling or an unknown factor that is biologically plausible. And lastly, they state that it might not be useful in all situations and null hypothesis testing should be preferred.








…wait what? I thought you were telling us that null-hypothesis tests were terrible and model selection infinitely better? Perhaps they have a little realistic sense about this model after all.

These models do seem like they could be extremely useful, but I would exercise extreme caution about making too many post-hoc conclusions based on the data. And while these models are evaluated as best-fit, there is no objective decision for when the best is good ENOUGH. All models could be relatively terrible at evaluating the natural situation. I would see Anderson and Burnham’s paper for more discussion about the dangers of these tests.