Also called confidence.

The level of certainty that the A/B test results achieved are accurate and not just a result of random chance or error.

When declaring a winner, you want to be fully confident that your findings are accurate and you’re not committing a type I error.

Confidence level is calculated using the equation 1 - α where α = the significance level. A result is significant when the p-value is ≤0.05.

Confidence level and significance level, expressed through p-value, are interdependent.

The higher the confidence level, the lower the significance level, and the less likely the result occurred due to random chance. 

And vice versa.

Or visually written:

↑ confidence level, ↓ p-value, ↑ statistical significance. 

And conversely:

↓ confidence level, ↑ p-value, ↓ statistical significance.

As a general best practice, you should aim to achieve a minimum 95% level of confidence.

A 95% confidence level means you can be confident 95 times out of 100 that you haven’t made a type I error. There’s just a 5% chance the results are due to random factors

On GuessTheTest, we aim to publish only tests that have obtained 95% confidence or above.

Note, it’s impossible to reach 100% confidence. You can only ever achieve 99.99% confidence.