Also called confidence.
The level of certainty that the A/B test results achieved are accurate and not just a result of random chance or error.
When declaring a winner, you want to be fully confident that your findings are accurate and you’re not committing a type I error.
Confidence level and significance level, expressed through p-value, are interdependent.
The higher the confidence level, the lower the significance level, and the less likely the result occurred due to random chance.
And vice versa.
Or visually written:
↑ confidence level, ↓ p-value, ↑ statistical significance.
↓ confidence level, ↑ p-value, ↓ statistical significance.
As a general best practice, you should aim to achieve a minimum 95% level of confidence.
A 95% confidence level means you can be confident 95 times out of 100 that you haven’t made a type I error. There’s just a 5% chance the results are due to random factors
On GuessTheTest, we aim to publish only tests that have obtained 95% confidence or above.
Note, it’s impossible to reach 100% confidence. You can only ever achieve 99.99% confidence.« Back to Glossary Index