p-Value
If a friend flips a coin and gets heads 10 times in a row, it is the number answering: 'Assuming no cheating, what are the chances of such a pure coincidence?'
Definition A p-value (probability value) is a statistical metric between 0 and 1 that indicates the likelihood that an observed experimental result occurred 'purely by chance.' A very small p-value means it is extremely unlikely that the outcome happened solely due to random fluke under the assumption that no real effect exists.
Let's Start with a Coin Toss Bet
Imagine making a coin toss bet with a friend, and their coin lands on heads 10 times in a row. Naturally, you would start to strongly suspect, 'Did they rig this coin?'
With a fair coin, the probability of getting 10 consecutive heads is only about 0.1%. A p-value is simply the calculated number for the probability of seeing such an extreme result, under the baseline assumption that 'the coin is perfectly normal.'
If the p-value of this coin toss is 0.001 (0.1%), it means there is only a 1-in-1,000 chance that this happened purely by coincidence without any trickery. When the probability of random chance is that absurdly low, common sense leads us to ditch the coincidence hypothesis and conclude our friend is cheating.
Plant Fertilizer Experiments and the 0.05 Cutoff
Suppose you feed plants a new eco-friendly fertilizer, and they grow much faster than pots given only water. You now need to figure out whether the plants thrived because of the fertilizer or simply because those seeds happened to be naturally stronger.
Scientists test hypotheses statistically in situations like this. By convention, researchers consider a result statistically significant when the chance of it being a fluke is less than 5%โmeaning a p-value below 0.05. This threshold is called the significance level.
If the fertilizer experiment yields a p-value of 0.02 (2%), it means there is only a 2% chance that the plants grew this well purely by random luck despite the fertilizer having zero effect. Because this is well below the 5% cutoff, researchers can reject the coincidence explanation and claim credit for the fertilizer's growth-boosting effect.
To Be More Precise
To be more precise, a p-value is not the probability that a new finding or claim is true. It is strictly the conditional probability of obtaining data as extreme as observed, assuming the premise of 'no effect at all' (the null hypothesis) is true.
Moreover, a p-value below 0.05 does not automatically mean a product or discovery is a groundbreaking innovation. If you increase the sample size to tens of thousands of potted plants, even a tiny, negligible difference can produce a p-value below 0.05 and appear statistically significant.
Ultimately, while the p-value is a fantastic traffic light for filtering out wild flukes, it cannot guarantee real-world importance or absolute truth on its own. That is why modern researchers carefully examine effect sizes and confidence intervals alongside the p-value.
๐ค Common misconceptions
A p-value of 0.01 means there is a 99% probability that my hypothesis is true.
A p-value is not the probability that your hypothesis is correct. It is only the probability of observing such extreme data by pure chance, assuming there is no real effect.
๐งบ Where you meet it
A p-value measures the probability that a result happened purely by chance, and results below 0.05 are generally considered not to be coincidences.