Results: When’s the highest converting time to send an email?

Which landing page format won in December, but lost in January? Copy

  • Version A
  • Version B
Loading ... Loading ...

Difference Between Versions:

Version A – Email creative, sent at 8 a.m. Pacific Standard Time (PST)
Version B – Exact same email creative, but instead sent at 6 p.m. PST

 


Key Performance Indicator (KPI): 

Open Rate, Unique Click Rate, Revenue, Average Order Value (AOV)


Test Goal:

Increase the number of readers opening the email and purchasing auto part products

 

Traffic Source:

Existing subscriber email list

 

Audience: 

Car enthusiasts - both vehicle owners and mechanic shop owners

 

Organization: 

Auto Parts Warehouse - online auto parts and accessories store

Sign-up for the newsletter to get FREE A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test
This week’s Featured Test is brought to you by:

Test Run By

Zeta Global

Test Run For

Auto Parts Warehouse

Test Run On

Zeta Global

WINNING VERSION

A

Poll Results - The Best Guesses:

Which landing page format won in December, but lost in January? Copy

  • Version A
  • Version B
Loading ... Loading ...

Ultimately, 8 a.m. was the winning time, but not across the board. . .

Read on to learn more about the effect of audience segmentation and email timing.


Test Details & Background:

To conduct this timely email test, Auto Parts Warehouse, an online auto parts and accessories e-tailer, teamed up with the data-driven marketing firm, Zeta Global.

The test goal was to determine the best day and time to send a promotional email to Auto Parts Warehouse subscribers – a large group of car enthusiasts and auto mechanics.


Hypothesis:

The testing team suspected email send time would impact conversions, but wasn’t sure when would work best. So, decided to test.


 

Test Set-up:

Testing took place over a period of 9 weeks. During this time, 1.8 million readers received 8 promotional emails.

Of the 1.8 million net email subscribers mailed, traffic was split into 4 equal groups of 450 thousand readers.

Every week, each group received the same message, but at different times, or on different days. The email send times were broken down as follows:

  • Control Group — 8 a.m.
  • Group B — 10 a.m.
  • Group C — 2 p.m.
  • Group D — 6 p.m.

All times were in Pacific Standard Time (PST).

Emails were sent on Sunday, Monday, Wednesday, or Friday.

For example, on 1 week, the control group received this email offer on Wednesday at 8 a.m, PST:

Existing email list of subscribers

Groups B and C received the same email on the same day, but the message came into their inboxes at 10 a.m. and 2 p.m. PST, respectively.

Group D also saw the same message, but received it Wednesday at 6 p.m., PST. An example email campaign looked like this:

versionb-6pm

The test ran on Zeta Global’s proprietary email platform.

Open rates, click rates (unique clicks over net messages delivered), revenue per net messages delivered, and Average Order Value (AOV) were tracked across all email sends.

Results were further broken down by whether the viewer was on a desktop or mobile device.


The Real-Life Results:

Winner: Version A – the message sent out at 8 a.m. won. Sort of. . . Results varied by the conversion metric tested, and the time of day.

 

Open rate

Looking at open rate by time and day, 6 p.m. was actually the best time to send an email everyday — except Wednesday, when 10 a.m. had the highest open rate.

Open rates were lowest at 8 a.m., every day of the week.

 

Click rate

Click rate was the number of unique clicks over net messages delivered. A click occurred anytime a reader clicked on a link within the email newsletter.

Like open rates, click rates were also highest at 6 p.m. and lowest at 8 a.m., every day of the week except Sunday (when 2 p.m. was best.)

However, Monday at 6 p.m. was the absolute best time for click rates.

The lowest click rates were on Friday at 8 a.m.

 

Revenue and conversion rate

Initially, revenue and conversion rates were not statistically significant. So, no valid conclusions could be drawn. But, the testing team ran a second validation test looking at an isolated data set.

For this data set, they parsed out the findings, narrowing down results only for 8 a.m. versus 6 p.m. conversions.

The findings were interesting.

Even though unique open and click rates were highest at 6 p.m., sales were actually highest at 8 a.m. on Wednesday.

 

Conversion rates by platform

To further understand the results, the team segmented data by mobile versus desktop users. Conversion rates were similar across desktop and mobile devices.

Unique open rates and click rates were highest at 6 p.m. But, strangely, revenue was highest at 8 a.m.

Even stranger, although 11.1% more purchases were made at 6 p.m., Average Order Value (AOV) was 15.4% higher at 8 a.m.

This trend showed more expensive products were bought in the morning, even if less people purchased them.

 

Overall findings

The emails sent out at 6 p.m. achieved the highest open and click rates, most days of the week. (So, if you guessed Version B, give yourself a pat on the back).

But. . . total sales were highest when the message was sent out at 8 a.m.

The 8 a.m. Wednesday send time was, therefore, declared the ultimate winner because it had the biggest impact on the bottom line.

Results achieved 95% confidence.


How Trustworthy Are The Results?

Given the fact the sample size was very large (450 thousand readers per group), the test ran for an adequate time period (9 — which is very long considering most email tests run just 1 time), and results achieved 95% confidence, it’s highly likely the study findings are valid.

However, it’s important to note, that even with this scientific rigour, initial revenue and conversion results were not statistically significant. It was not until the testing team ran a second validation test, looking at an isolated data set, that meaningful results were derived.

This trend shows the the importance of parsing down key slices of your most meaningful data.


Analysis:

Why did the 6 p.m. send time out perform on most accounts, but fall short when it came to money? There are likely a couple reasons:

1. Timing may be everything, but not when it comes to email conversions

As we saw with this seasonality test, different audiences have different needs and different mindsets.

In the case of email sends, you’d think a higher number of email opens and clicks at 6 p.m. would translate to more high-dollar purchases at this time. But, that wasn’t the case.

The reason: different consumers were making different purchase decisions at different times.

At 6 p.m., car enthusiasts were casually checking their email and clicked to see what kind of deals they could get. But, weren’t necessarily committed to buying. And, those who did buy were purchasing less expensive, lower-end items.

However, at 8 a.m. the picture was much different. According to Zeta Global, the majority of users, at this time, were small business owners, like mechanics or shop owners, looking to purchase auto parts to fix clients’ vehicles. These visitors had high conversion intent and were willing to make more expensive purchases.

These results show, sending an email that converts requires more than just strategically timing your message. You must also understand who your audience is, what their needs are, and what they’ll respond to best, when. Then, segment your messages, and send times, accordingly.

2. You need to dig deeper into the funnel to get the most accurate picture

It’s not always enough to take results at face value. Sometimes, you need to dig deeper into the conversion funnel to get a true picture of what’s happening.

Had the testing team just looked at open rates and click rates, they would have assumed the 6 p.m. send time was the way to go.

However, they took their findings one step further and assessed the effect open and click rates had on revenue and AOV. In going deeper into their analysis — and the funnel — the team spotted an interesting trend: increased opens and clicks didn’t equate to increased purchase amounts.

With sales being the most important metric, the 8 a.m. send time was the real cash cow.


Tangible Takeaway & Immediate Application

There is no magic date or time for a high-converting email send. The best time is determined by testing, and targeting, specific audience segments.

When analyzing results, consider not only conversion rates across the funnel, but also who your audience is, what their needs are, and what they’ll respond to best, when. Then, segment your messages, and send times, accordingly.


What Do You Think?

What are your thoughts? Share your comments in the section below.

Poll Results - The Best Guesses:

Which landing page format won in December, but lost in January? Copy

  • Version A
  • Version B
Loading ... Loading ...

Sign-up for the newsletter to get FREE gamified A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test

3
COMMENTS: Tell Us Your Thoughts.

avatar
2 Comment threads
1 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
2 Comment authors
Deborah OMarissa Ryan Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Marissa Ryan
Guest

I find the results of this test really interesting! I think the industry this test is taking place in also affects the results. I ended up guessing “B” as the winner, based on other e-commerce tests that I’ve run myself in which the just-after-work timing was more effective than morning timing. Most of those high performers were for what I would consider ‘personal’ shopping, such as fashion, beauty, homewares, gifts, etc. This was for a more ‘industrial’ type of product. I always try to think about the factors at play and specifics like the user intent of purchasing large items… Read more »

Other Tests You Might Enjoy

Test Run By

Zeta Global

Test Run For

Auto Parts Warehouse

Test Run On

Zeta Global