Difference Between Versions:
Version A – Email creative, sent at 8 a.m. Pacific Standard Time (PST)
Version B – Exact same email creative, but instead sent at 6 p.m. PST
Key Performance Indicator (KPI):
Average order value
Increase the number of readers opening the email and purchasing auto part products
Existing subscriber email list
Car enthusiasts - both vehicle owners and mechanic shop owners
Auto Parts Warehouse - online auto parts and accessories store
Sign-up for the newsletter to get FREE A/B tests and more valuable content delivered to you every other week.
Become a More Successful Digital MarketerPro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage
This week’s Featured Test is brought to you by:
Poll Results - The Best Guesses:
Ultimately, 8 a.m. was the winning time, but not across the board. . .
Read on to learn more about the effect of audience segmentation and email timing.
Test Details & Background:
The test goal was to determine the best day and time to send a promotional email to Auto Parts Warehouse subscribers – a large group of car enthusiasts and auto mechanics.
The testing team suspected email send time would impact conversions, but wasn’t sure when would work best. So, decided to test.
Testing took place over a period of 9 weeks. During this time, 1.8 million readers received 8 promotional emails.
Of the 1.8 million net email subscribers mailed, traffic was split into 4 equal groups of 450 thousand readers.
Every week, each group received the same message, but at different times, or on different days. The email send times were broken down as follows:
- Control Group — 8 a.m.
- Group B — 10 a.m.
- Group C — 2 p.m.
- Group D — 6 p.m.
All times were in Pacific Standard Time (PST).
Emails were sent on Sunday, Monday, Wednesday, or Friday.
For example, on 1 week, the control group received this email offer on Wednesday at 8 a.m, PST:
Groups B and C received the same email on the same day, but the message came into their inboxes at 10 a.m. and 2 p.m. PST, respectively.
Group D also saw the same message, but received it Wednesday at 6 p.m., PST. An example email campaign looked like this:
The test ran on Zeta Global’s proprietary email platform.
Open rates, click rates (unique clicks over net messages delivered), revenue per net messages delivered, and Average Order Value (AOV) were tracked across all email sends.
Results were further broken down by whether the viewer was on a desktop or mobile device.
The Real-Life Results:
Winner: Version A – the message sent out at 8 a.m. won. Sort of. . . Results varied by the conversion metric tested, and the time of day.
Looking at open rate by time and day, 6 p.m. was actually the best time to send an email everyday — except Wednesday, when 10 a.m. had the highest open rate.
Open rates were lowest at 8 a.m., every day of the week.
Click rate was the number of unique clicks over net messages delivered. A click occurred anytime a reader clicked on a link within the email newsletter.
Like open rates, click rates were also highest at 6 p.m. and lowest at 8 a.m., every day of the week except Sunday (when 2 p.m. was best.)
However, Monday at 6 p.m. was the absolute best time for click rates.
The lowest click rates were on Friday at 8 a.m.
Revenue and conversion rate
Initially, revenue and conversion rates were not statistically significant. So, no valid conclusions could be drawn. But, the testing team ran a second validation test looking at an isolated data set.
For this data set, they parsed out the findings, narrowing down results only for 8 a.m. versus 6 p.m. conversions.
The findings were interesting.
Even though unique open and click rates were highest at 6 p.m., sales were actually highest at 8 a.m. on Wednesday.
Conversion rates by platform
To further understand the results, the team segmented data by mobile versus desktop users. Conversion rates were similar across desktop and mobile devices.
Unique open rates and click rates were highest at 6 p.m. But, strangely, revenue was highest at 8 a.m.
Even stranger, although 11.1% more purchases were made at 6 p.m., Average Order Value (AOV) was 15.4% higher at 8 a.m.
This trend showed more expensive products were bought in the morning, even if less people purchased them.
The emails sent out at 6 p.m. achieved the highest open and click rates, most days of the week. (So, if you guessed Version B, give yourself a pat on the back).
But. . . total sales were highest when the message was sent out at 8 a.m.
The 8 a.m. Wednesday send time was, therefore, declared the ultimate winner because it had the biggest impact on the bottom line.
Results achieved 95% confidence.
How Trustworthy Are The Results?
Given the fact the sample size was very large (450 thousand readers per group), the test ran for an adequate time period (9 — which is very long considering most email tests run just 1 time), and results achieved 95% confidence, it’s highly likely the study findings are valid.
However, it’s important to note, that even with this scientific rigour, initial revenue and conversion results were not statistically significant. It was not until the testing team ran a second validation test, looking at an isolated data set, that meaningful results were derived.
This trend shows the the importance of parsing down key slices of your most meaningful data.
Why did the 6 p.m. send time out perform on most accounts, but fall short when it came to money? There are likely a couple reasons:
1. Timing may be everything, but not when it comes to email conversions
As we saw with this seasonality test, different audiences have different needs and different mindsets.
In the case of email sends, you’d think a higher number of email opens and clicks at 6 p.m. would translate to more high-dollar purchases at this time. But, that wasn’t the case.
The reason: different consumers were making different purchase decisions at different times.
At 6 p.m., car enthusiasts were casually checking their email and clicked to see what kind of deals they could get. But, weren’t necessarily committed to buying. And, those who did buy were purchasing less expensive, lower-end items.
However, at 8 a.m. the picture was much different. According to Zeta Global, the majority of users, at this time, were small business owners, like mechanics or shop owners, looking to purchase auto parts to fix clients’ vehicles. These visitors had high conversion intent and were willing to make more expensive purchases.
These results show, sending an email that converts requires more than just strategically timing your message. You must also understand who your audience is, what their needs are, and what they’ll respond to best, when. Then, segment your messages, and send times, accordingly.
2. You need to dig deeper into the funnel to get the most accurate picture
It’s not always enough to take results at face value. Sometimes, you need to dig deeper into the conversion funnel to get a true picture of what’s happening.
Had the testing team just looked at open rates and click rates, they would have assumed the 6 p.m. send time was the way to go.
However, they took their findings one step further and assessed the effect open and click rates had on revenue and AOV. In going deeper into their analysis — and the funnel — the team spotted an interesting trend: increased opens and clicks didn’t equate to increased purchase amounts.
With sales being the most important metric, the 8 a.m. send time was the real cash cow.
Tangible Takeaway & Immediate Application
There is no magic date or time for a high-converting email send. The best time is determined by testing, and targeting, specific audience segments.
When analyzing results, consider not only conversion rates across the funnel, but also who your audience is, what their needs are, and what they’ll respond to best, when. Then, segment your messages, and send times, accordingly.
What Do You Think?
What are your thoughts? Share your comments in the section below.