All About Bayesian: A Look At A New Statistical Model For A/B Testing
By: Deborah O’Malley| 2019
Bayesian Statistics isn’t new. It’s been around for a while. Since the 1770’s, in fact.
But, its just now making inroads as computing capabilities improve and marketers become more savvy about proper A/B testing techniques. Case in point, A/B testing software company, Visual Website Optimizer (VWO), recently started using Bayesian Statistics in their new SmartStats A/B testing engine.
What is Bayesian Statistics?
Bayesian statistics is a statistical model that can be applied to A/B testing to more accurately determine the precise range of your conversion rate, answering the question: “What’s the probability of version B beating A?”
Named after the British mathematician, Thomas Bayes, the model is based on “Bayes Theorem,” also called Bayes’ Rule. The theorem is expressed with this formula:
Breaking down the formula:
- A and B are considered events
- P(A|B) is the conditional probability of A occurring, given B is true
- P(B|A) is the conditional probability of B occurring, given A is true
- P(A) is the prior probability A will occur
- P(B) is the prior probability B will occur
Is your head spinning yet? Stick with it, it’ll all makes sense in a minute. . .
First though, you need to accept the mindset that, in Bayesian reasoning, you can continually update your beliefs about data, as your gather evidence. In A/B testing, you gather evidence by running a test. After observing the evidence, your opinions may change. This is the idea behind probability.