By: Deborah O'Malley, M.Sc | Last updated December, 2021

How long do you need to run your A/B test to achieve valid results?

In true Conversion Rate Optimization (CRO) style, the answer is. . . it depends. . .

What does it depend on?

The short answer is: everything.

The longer answer is: a myriad of factors including, but not limited to:

• The type of test you're running
• How many variants you're testing
• Seasonal factors
• Sales cycles

Let's dig a bit deeper into each of these elements.

What A/B test timing depends on:

How long you should run an A/B test depends on several factors, including, but not limited to:

The type of test you’re running:

• An email test --> may run only for a few hours to a few days because the email is typically just sent once and most users will open the email and convert – or not – within a limited timeframe.

• A website test --> doesn’t usually have a limited send or open time so may need to last much longer.

How many variants you’re testing:

• The more variants you test, the more traffic you need. You don’t want to spread your traffic too thin.
  
• In situations with lower traffic, it's best to limit the number of variants tested, usually to 2 (A vs. B) so each version receives enough traffic to draw conclusive results in an adequate timeframe.

Seasonal factors:

• All audiences are different and may exhibit different buying patterns throughout the year.
  
• This real-life GuessTheTest case study provides an important example of how seasonality can effect conversions, especially over periods, like the Christmas holiday season, when shoppers are buying gifts for others – not themselves.

Sales cycles:

• For some businesses, or industry verticals, traffic may follow distinct buying patterns. A sale may happen over several steps or stages. And you may need to nurture leads for a long time before they convert.
  
• Taking this cycle into account is important to yield accurate testing data.

How long is “long enough” to run a valid A/B test?

Because every website is different, there’s no set time period for how long a test should run.

The right answer is: long enough to accurately take into account factors like seasonality and your company’s sales cycles.

What is the industry best practice?

However, taking all these factors into account, there is a general A/B testing best practice.

The best practice is to let a test run for a minimum of 2-weeks but no longer than 6-8 weeks.

This time period is selected so that any trends observed over a one-week period, or less, can be confirmed and validated over again.

For example, if users behave differently on the weekend, you’ll want to see that same pattern repeat across two weekends, rather than just once, to smooth out any questions or discrepancies in the data.

However, if you need to run a test for longer than 6 weeks, you likely don’t have enough traffic to get a significant result.

As well, more than about 6 weeks and the data starts to become muddied. Things like user patterns may shift or cookies become deleted, introducing a whole new set of variables into the equation.

And, as a result, you won’t know if it’s changing user behavior or something else that’s contributing the test results.

Plus, it can be expensive or frustrating when tests need to run for several months.

So, if you have to run your test longer than the ideal 2-6 week timeframe, you should question how worthwhile it is to run the test.

Testing duration calculator

A testing duration calculator, like this one, can help determine how long it will likely take for a test to conclude, based on projected site traffic.

Previously collected analytics data can give you advanced insight into traffic trends.

What if I don’t have enough traffic to run a valid test in less than 6 weeks?

Don’t fret.

If you have the time, resources, and budget, running a low-traffic A/B test is still better than not testing anything at all.

A low traffic test that takes longer to run will still give you some indication of how site visitors are likely to act and how a variant will most probably perform.

Results, however, are more of an approximation of what is likely to work rather than absolute evidence of what has proven to work.

So, keep this outcome in mind when deciding to implement any so-called “winning” designs.

Your thoughts?

Hope you found this article helpful! If so, please comment and share widely.

Do you have any questions or thoughts? Give your feedback in the comments section below: