A couple of days ago I was trying to nail down p-value, significance levels and null hypotheses. I know that if the p-value is lower than 0.05, the result is significant and the null hypothesis is rejected – but that’s just memorization. I wanted to understand it, know when it should be applied, and be able to explain it to others. I turned to my trusty co-worker, R., to help in this endeavor. (If you prefer you can skip the dorky “Brianne comes to understand p-values” conversation and go right to the “why this shit is important” section below.)
Me: R – do you have a good understanding of the concepts underlying p-values, significance levels and null hypotheses?
R: Not really a great understanding. I use stats software to calculate p-values. Anything less than 0.05 is significant.
Me: Yeah, I know that, but can you help me understand something? C’mere and look at my screen.
[R, who is used to me pulling us off-topic, sighs good-naturedly and comes over]
Me: So look at this the bell curve in Wikipedia:
R: I don’t think it’s arbitrary.
Me: Well, it’s not set by …err… &*^% … I don’t know. Let’s tackle that one another day. So, let’s say we set it at 0.05 (me mumbling under my breath “arbitrarily”). That means 95% of the time (everything not green) we’re not going to observe the null hypothesis?
R: I hate double negatives.
R: No. It’s saying that there is a 5% probability of making our observation purely by chance. The smaller the number the less likely it is that our observation was made by chance. The greater the number the more likely it was made by chance. A small number rejects the null hypothesis.
Me: Okay. It’s an estimate of how likely we are to observe our result by chance. So…let’s say that we’re trying to determine if a chemical is stable over time. We test it several times over a month, then look at the difference over time in how the chemical performs. We do a linear regression and get a slope of 0.1* The p-value on the data is calculated to be 0.05. That means there’s a 5% probability of obtaining the observed result (0.1) or greater by chance if no real effect exists (that is, if our chemical is stable).
A long pause.
Me: Sweet. I think I get it. But, that means that p-value is sort of the opposite of what we observe…
R: Yeah, it’s counter-intuitive.
Me: It’s math. It’s all counter-intuitive to me.
Biodork: helping you lose confidence in the scientists behind the development of the technologies you use every day!
I’m kidding. Statistics is not what I do for my company. But understanding statistical analyses is important in my field and it feels good to finally have a grasp of the concept.
I was a little hesitant to write this because it feels like p-value is something that should be understood by A Real ScientistTM. You see p-values all over the literature, so of course everyone understands them, right? But it’s not an easy concept. It is counter-intuitive. And I thought I’d throw it out there that I for one was struggling with it, and it’s okay if you are too. Mathphobes unite!
There’s another good reason to understand p-values and statistical analyses in general – It’s a huge part of not getting fooled by shoddy statistics put out by proponents of bullshit. I’m looking at you, alternative medicine. Some readers of this blog are probably also fans of Science-Based Medicine. SBM has many articles that discuss the statistics gymnastics that alt med proponents perform to make it seem as if their [insert bullshit product here] has an effect on [insert condition here].
I’m reading a book right now called Intuitive Biostatistics by Dr. Harvey Motulsky. I’m finding it hard to put down, and I don’t often say that about math books. Here’s how Dr. Motulsky explains his book:
Unlike statistics texts that emphasize mathematics, Intuitive Biostatistics focuses on proper scientific interpretation of statistical tests. The book is perfect for researchers, clinicians and students who need to understand statistical results published in biological and medical journals. Intuitive Biostatistics covers all the topics typically found in introductory texts, and adds several topics of particular interest to biologists – including multiple comparison tests, nonlinear regression and survival analysis.
The first chapter is all about how we trick ourselves into seeing patterns where none exist, and the importance of correctly analyzing data so that it can speak for itself without our dumb brains getting in the way. Motulsky gives illuminating examples throughout the book and leaves the math formulae to the statistics textbooks. You can find it free online or on Amazon if you want a paper copy. I recommend this book for every skeptic who has to deal with data sets or statistical defenses of woo.
And finally, if you have any rules or tricks to help explain p-value in a concise manner, I’d love to see them in the comments below. Also, I am a self-admitted mathphobe and doubter of my own mathematical skills, so if you catch me in an error, please do call it out. I’ll be happy to learn and grateful for the chance to clear up any misinformation that I might be spreading.
*For my mathphobe friends – a simple explanation for a linear regression is that it’s a line drawn through the points that we’ve just plotted, and a slope refers to…well…the slope of the line we’ve drawn. A slope of 0.1 in the example means there is about a 10% difference between how the chemical performed on the last day as compared to how it performed on the first day we tested it.
Le hand-drawn by moi linear regression.