There’s some bad news and good news.
Let’s get the bad news out of the way. Then, go into the good news.
The bad news is: what you’ve been taught about how to calculate a valid A/B test is probably wrong!
That cause you’ve probably been taught using Frequentist Statistics.
While there’s nothing wrong with Frequentist Statistics, the model doesn’t work that well when applied to A/B testing.
In Frequentist statistics, the only way to validly address this question is by stating a null hypothesis. And here’s where it gets more mind boggling. . .
Unlock Access. Sign up to become a Pro Member. Get complete access to this helpful content, plus so much more.
Calculating the Minimum Detectable Effect (MDE) can feel like a hair-pulling, speculative exercise. Luckily, this step-by-step guide is here to show you exactly how to accurately calculate your MDE to yield powerful, trustworthy test results.
Low sample testing is problematic and can result in hugely mis-leading data where tests appear to be enormous winners but, in actuality, aren't. This article presents a practical 5-step approach to overcome the traps of low sample testing and arrive at valid, trustworthy results. What are the 5-steps? Find out here.
Understanding and calculating statistical significance is quite complex. Many experimenters don’t truly know what statistical significance is or how to derive a statistically significant test result. To properly call a winning (or losing) A/B test, it’s really important to clearly understand what a statistically significant result is and means. This article, written in plain English is here to set it all straight for you.