Officially defined, p-value is: the probability of the test producing an observed result at least as or more extreme than the ones observed in your data, assuming the null hypothesis is true.

In plain English, this concept means: p-value tells you how likely it is that the outcome occurred, if the null hypothesis is true.

The null hypothesis assumes there is no difference in conversion rates.

So, if, on the unlikely chance you detect a difference in conversion rates, you use p-value to tell you the probability there’s actually, really a real difference between variants not just due to error or random chance.

This probability is expressed as a value. The value is directly tied into significance level alpha (α). When the p-value is less than alpha (usually set a < 0,05), you reject the null hypothesis.

In rejecting the null hypothesis, you accept the alternative: that there is truly a conversion difference between variants not just due to random chance or error. 

Hallelujah! You've found a winner.

An unusual and surprising outcome, the result is considered significant.

Therefore, a p-value of ≤0.05 means the result is deemed statistically significant.

However, while a p-value of ≤0.05 is, typically, accepted, many data purists will balk at this threshold. They’ll tell you a significant test should have a p-value ≤0.01. And some data scientists even argue it should be lower.

However low you go, it seems everyone can agree: the closer the p-value to 0 the stronger the evidence the conversion lift is real – not just the outcome of random chance or error. 

Which, in turn, means there’s a higher degree of confidence probability you’ve actually found a winner. 

Or written another way: ↓ p-value, ↑ significant the finding.