In frequentist statistics, your aim is to prove or disprove the null hypothesis.
The null hypothesis assumes there is no difference between versions.
When you carry out an A/B test using frequentist statistics, your aim is to prove or disprove the null hypothesis.
The null hypothesis is disproven if a winning test result emerges showing a conversion difference between versions.
Although typically used in most A/B testing platforms, the drawback of the frequentist method is it requires a large sample size and a valid testing timeframe in order to declare statistically significant results with a high level of confidence.
In contrast, the Bayesian methodology enables testers to run an experiment with a much smaller sample size within a much shorter testing timeframe.
The tradeoff is a sacrifice of statistical rigour only possible through a properly carried out hypothesis-based A/B test experiment.
The reason is because a test run with the Bayesian method states only the likely probability the outcome will occur.« Back to Glossary Index