By: Deborah O'Malley, M.Sc | Last updated September, 2021
In true Conversion Rate Optimization (CRO) style, the answer is. . . it depends. . .
What does it depend on?
The short answer is: everything.
The longer answer is: a myriad of factors including, but not limited to:
Let's dig a bit deeper into each of these elements.
How long you should run an A/B test depends on several factors, including, but not limited to:
Because every website is different, there’s no set time period for how long a test should run.
The right answer is: long enough to accurately take into account factors like seasonality and your company’s sales cycles.
However, taking all these factors into account, there is a general A/B testing best practice.
The best practice is to let a test run for a minimum of 2-weeks but no longer than 6-8 weeks.
This time period is selected so that any trends observed over a one-week period, or less, can be confirmed and validated over again.
For example, if users behave differently on the weekend, you’ll want to see that same pattern repeat across two weekends, rather than just once, to smooth out any questions or discrepancies in the data.
However, if you need to run a test for longer than 6 weeks, you likely don’t have enough traffic to get a significant result.
As well, more than about 6 weeks and the data starts to become muddied. Things like user patterns may shift or cookies become deleted, introducing a whole new set of variables into the equation.
And, as a result, you won’t know if it’s changing user behavior or something else that’s contributing the test results.
Plus, it can be expensive or frustrating when tests need to run for several months.
So, if you have to run your test longer than the ideal 2-6 week timeframe, you should question how worthwhile it is to run the test.
A testing duration calculator, like this one, can help determine how long it will likely take for a test to conclude, based on projected site traffic.
Previously collected analytics data can give you advanced insight into traffic trends.
If you have the time, resources, and budget, running a low-traffic A/B test is still better than not testing anything at all.
A low traffic test that takes longer to run will still give you some indication of how site visitors are likely to act and how a variant will most probably perform.
Results, however, are more of an approximation of what is likely to work rather than absolute evidence of what has proven to work.
So, keep this outcome in mind when deciding to implement any so-called “winning” designs.
Hope you found this article helpful! If so, please comment and share widely.
Do you have any questions or thoughts? Give your feedback in the comments section below: