All About Bayesian: A Look At A New Statistical Model For A/B Testing

By: Deborah O’Malley| 2019

Bayesian Statistics isn’t new. It’s been around for a while. Since the 1770’s, in fact.

But, its just now making inroads as computing capabilities improve and marketers become more savvy about proper A/B testing techniques. Case in point, A/B testing software company, Visual Website Optimizer (VWO), recently started using Bayesian Statistics in their new SmartStats A/B testing engine. 

What is Bayesian Statistics?

Bayesian statistics is a statistical model that can be applied to A/B testing to more accurately determine the precise range of your conversion rate, answering the question: “What’s the probability of version B beating A?”

Named after the British mathematician, Thomas Bayes, the model is based on “Bayes Theorem,” also called Bayes’ Rule. The theorem is expressed with this formula:

Breaking down the formula:

  • A and B are considered events
  • P(A|B) is the conditional probability of A occurring, given B is true
  • P(B|A) is the conditional probability of B occurring, given A is true
  • P(A) is the prior probability A will occur
  • P(B) is the prior probability B will occur

Is your head spinning yet? Stick with it, it’ll all makes sense in a minute. . .

First though, you need to accept the mindset that, in Bayesian reasoning, you can continually update your beliefs about data, as your gather evidence. In A/B testing, you gather evidence by running a test. After observing the evidence, your opinions may change. This is the idea behind probability.

Probability Defined

Only Pro Members have access to this post. Sign up to become a Pro Member and gain access to this valuable content, plus so much more.

Learn More.