Results: Which email subject line won?

Warren, Save 85-90% Compared To Other Marketing Conferences

Click here to enlarge

Warren, Get Your A/B Testing Summit All-Access Pass

Click here to enlarge

Which email subject line won?

Loading ... Loading ...

Difference Between Versions:

Version A – <Name>, Save 85-90% Compared To Other Marketing Conferences
Version B – <Name>, Get Your A/B Testing Summit All-Access Pass


Key Performance Indicator (KPI): 

Email open rate


Test Goal:

Increase the number of people opening the email and purchasing all All-Access Passes

 

Traffic Source:

Existing email list of registrants who had signed up for the A/B Testing Summit

 

Audience: 

Digital marketers interested in A/B testing and optimization

 

Organization: 

A/B Testing Summit - Online virtual event to help digital marketers

Sign-up for the newsletter to get FREE A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test
This week’s Featured Test is brought to you by:

Test Run By and For

A/B Testing Summit

Test Run On

ActiveCampaign

WINNING VERSION

B

Poll Results - The Best Guesses:

Which email subject line won?

  • Version A
  • Version B
Loading ... Loading ...

Test Details & Background:

The A/B Testing Summit is an online virtual event that helps marketers optimize their digital experiences and grow revenue. At this year’s 2019 Summit, over 30 optimization experts shared their knowledge, during a jam-packed three-day online event.

When the Summit finished, the recorded talks were put behind a paywall. An All-Access Pass was made available, providing complete access to all the talks, plus over 70 audio and video recording from experts like Tim Ash, Flint McGlaughlin, and Brian Massey.

To announce pass availability, the Summit testing team crafted an email and decided to test which email subject line would convert best.

Previous testing revealed personalizing the subject line, with the registrant’s first name, was most effective at increasing email open rates.

So, personalization was incorporated into this test. However, the team was uncertain which subject line approach would outperform: one that emphasized savings – or gave a direct, clear Call To Action (CTA).

The audience – digital marketers interested in A/B Testing and optimization – had already received several emails about the Summit event. So, this email needed to catch attention, and offer benefit.


 

 

Hypothesis:

The testing team suspected the personalized email emphasizing the large savings, of 85-90%, would appear most beneficial, increasing email open rates, and subsequent All-Access Pass purchases.

However, the team also acknowledged these numeric figures might come across as confusing, or inauthentic. And, therefore, wouldn’t convert as well as the subject line with the direct, clear CTA.

So, decided to test.


Test Set-up:

To determine which subject line converted best, a multi-armed bandit test was set-up and run on the email marketing platform, ActiveCampaign.

The test ran for one day/one email send, and was sent to a total of 3,496 people.

As a multi-armed bandit test, only 30% of the email list was initially tested. This list was split 15% to version A, and 15% to version B.

As a winner emerged, the remaining 70% of the email list was tested, with more traffic automatically directed to the winning version.

As a result, just 524 readers saw the losing subject line. It said: <First Name> Save 85-90% Compared To Other Marketing Conferences

The winning version was sent to 2,972 readers. It said: <First Name>, Get You’re a/B Testing Summit All-All Pass.

With both subject lines, readers clicked to open the exact same email message, which appeared like this:

ab_testing_summit-sm

Email open rates and click rates were tracked across both versions.


The Real-Life Results:

Winner: Version B – the clear, directive Call To Action (CTA) clearly won.

Contrary to the hypothesis emphasizing savings did not work best.

Rather, being directive by telling registrants to get the All-Access Pass increased open rates 61.49%. Results achieved 99% confidence.

There was no statically significant difference in click rates, once the email was opened. Likely because the email copy was the same across both versions.

Significance was calculated using a calculator from AB Testguide. This screenshot provides a more detailed breakdown of the results:

test_results


How Trustworthy Are The Results?

This experiment was unusual because it was set-up as a multi-armed bandit test; more traffic was allocated to the winning version, as it emerged.

While this testing approach may seem unconventional, it’s considered a reliable and valid way to run tests and split traffic to yield quick, accurate results.

Given that the sample size was adequate (3,496 readers), the email test was run for a standard time period (1-day/1 email send), and results achieved 99% confidence, the test appears to hold strong validity.

However, it should be noted that multi-armed bandit testing isn’t always the best approach. As pointed out in this article, a drawback of multi-armed bandit testing is that, when conversion rates between variants are similar, the testing method is less efficient. Multi-armed bandit algorithms work best when the conversion rates are massively different (as in the case of this test, which achieved a 61.49% conversion lift).

But, since you can’t know — before testing — what the conversion difference will be, it’s best to use multi-armed bandit testing only when you’re in a time crunch, or you’re continuously optimizing on already proven results.


Analysis:

Why did the clear, directive CTA outperform?

There’s likely a few reasons:

1. Concreteness wins

Concreteness means stating pertinent facts and details and using the 5W’s and H (who, what, where, when, why, and, how) to answer all the questions in the reader’s mind.

On its own, stating marketers could save 85-90%, compared to other marketing conferences, wasn’t concrete.

The subject line didn’t detail how this number was calculated, or where this figure came from. Until readers clicked to open the email, there was no explanation about how this comparison was derived. With skepticism elicited, fewer readers clicked.

As well, the numbers of 85-90% savings may have seen too large to be authentic.

Or, may have created confusion because the numeric value was so large. As shown in this previously published GuessTheTest study, presenting smaller numeric formats may work best.

2. Clarity converts

While both email subject lines started with strong verbs: “save” or “get”, the verb “get,” coupled with the CTA, Get Your A/B Testing Summit All-Access Pass, was a lot clearer and more directive.

There was no ambiguity or room for misinterpretation. Readers knew what action to take. The only choice was whether or not to perform the action – and click to open the email – which many more readers chose to do.

This test provides yet another example that clarity converts.

3. You can’t trust your gut

This A/B test was set-up and run by a team of digital marketers with a lot of experience in A/B testing. Yet, the hypothesis was proven incorrect!

The results show, you really can’t trust your gut. You never truly know what will win – until you test.

Important disclosure – The A/B Testing Summit was co-organized by GuessTheTest. Reporting of the results is done in attempt to provide complete transparency and objectivity.


Tangible Takeaway & Immediate Application

Clarity converts. An email subject line with a clear, strong call to action provides direction that may help motivate readers to convert.

If you’d like to learn more about this important concept, get your All-Access Pass, to gain access to helpful, educational talks from over 30 optimization experts.


What Do You Think?

Why do you think the winning subject line outperformed? Share your thoughts in the comments section below.

Poll Results - The Best Guesses:

Which email subject line won?

  • Version A
  • Version B
Loading ... Loading ...

Sign-up for the newsletter to get FREE gamified A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test

5
COMMENTS: Tell Us Your Thoughts.

avatar
3 Comment threads
2 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
4 Comment authors
WarrenCliff SaundersDeborah Ovladmocanu Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
vladmocanu
Member
vladmocanu

This is really an interesting test. I’ll have to go and read more about multi-armed bandit test as it is the first time I hear this concept.
Thanks for sharing!

Cliff Saunders
Guest

I chose A.

I don’t recall a conscious reason.

Other Tests You Might Enjoy

Test Run By and For

A/B Testing Summit

Test Run On

ActiveCampaign