As statistical method of calculation used in traditional hypothesis-based A/B testing.

In frequentist statistics, your aim is to prove or disprove the null hypothesis.

The null hypothesis assumes there is no difference between versions.

When you carry out an A/B test using frequentist statistics, your aim is to prove or disprove the null hypothesis.

The null hypothesis is disproven if a winning test result emerges showing a conversion difference between versions.

Although typically used in most A/B testing platforms, the drawback of the frequentist method is it requires a large sample size and a valid testing timeframe in order to declare statistically significant results with a high level of confidence.

In contrast, the Bayesian methodology enables testers to run an experiment with a much smaller sample size within a much shorter testing timeframe.

The tradeoff is a sacrifice of statistical rigour only possible through a properly carried out hypothesis-based A/B test experiment.

The reason is because a test run with the Bayesian method states only the likely probability the outcome will occur.