Results: Which Facebook Ad Generated More Leads?

Halloween-themed Facebook ad

Click here to enlarge

Non Halloween-themed Facebook ad

Click here to enlarge

Which Facebook Ad Generated More Leads?

Loading ... Loading ...

Difference Between Versions:

Version A – Halloween-themed creative; featured orange and black color scheme with candy background
Version B – Non Halloween-themed creative; featured blue and white color scheme with polka dots


Key Performance Indicator (KPI): 

Leads, Link Clicks, Cost Per Mille (CPM), Cost Per Click (CPC), Clickthrough Rate (CTR)


Test Goal:

Increase the number of leads booking a test drive

 

Traffic Source:

Facebook users

 

Audience: 

Targeted Facebook ad users

 

Organization: 

Hyundai: Car company

Sign-up for the newsletter to get FREE A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test
This week’s Featured Test is brought to you by:

Test Run By

Iacoca Blue Advertising

Test Run For

Hyundai

Test Run On

Facebook

WINNING VERSION

B

Poll Results - The Best Guesses:

Which Facebook Ad Generated More Leads?

  • Version A
  • Version B
Loading ... Loading ...

The standard creative generated more leads (even though it lost on many other metrics)


Test Details & Background:

Iacoca Blue Advertising is a data-driven digital agency focused on creative advertising.

They conducted this tastefully treated Halloween Facebook ad creative test for their Canadian car client, Penticton Hyundai.

The test goal was to see if, during the month of October, the Halloween-themed creative would generate more leads, interested in booking a test drive.

With a successful A/B test result from a previous Halloween-themed Facebook ad experiment, the testing team wanted to verify if the holiday-themed creative would once again win.


 

Hypothesis:

Based on past experience, the agency expected the Halloween-themed ad would bring in more leads.

However, they also realized the spookily specific Halloween theme might not resonate that well with users throughout the entire month of October. So, decided to test what would work best.


Test Set-up:

To definitively determine which creative converted best, a Facebook ad A/B test was set-up and ran on Facebook, during the month of October.

The ad was broadly targeted.

The ad generated 23,362 impressions; traffic to the ad was split 50/50.

Half of viewers saw the colorful candy creative. It featured images of tempting treats in the background, with a Halloween-themed orange and black color scheme. The ad appeared like this:

version_a

The other half of Facebook ads users were shown the non Halloween-themed creative. It featured a fun polka dot background with a light and airy blue color scheme. It looked like this:

version_b

Leads clicking the “Learn More” button and filling out a booking form were tracked across both versions.

In addition, the team assessed the cost of the ads, both in terms of Cost Per Mille (CPM) and Cost Per Click (CPC).


The Real-Life Results:

Winner: Version B – contrary to the hypothesis, the non Halloween-themed creative won. But, not across all metrics.

—Cost Per Mille (CPM) and Cost Per Click (CPC)—

From a cost perspective, the Halloween-themed ad actually won. The Halloween-themed creative had a 6% lower CPM and 30% lower CPC rate, compared to the non Halloween-themed creative.

—Clickthrough rate (CTR)—

Looking at link clicks on the “Learn More” button, the Halloween-themed creative also had a 20% better CTR, compared to the standard creative.

—Leads–

But, despite the higher CTR, the halloween-themed creative actually generated 60% fewer leads than the regular ad. Now, that’s a scary stat!

It showed that, even though Facebook ad users were compelled to click on candy-colored creative, they weren’t ready to convert, or book a test drive.

Since the halloween-themed ad generated fewer leads — and the main objective was generating leads — the standard ad was declared the overall winner.


How Trustworthy Are The Results?

At the time of this publication, the final days of the study were wrapping up. Because the study hadn’t officially concluded within Facebook, a complete confidence level couldn’t be asserted.

However, given that the study ran for an adequate time period (nearly 1 month), and received a large number of impressions (23 thousand), it’s likely results hold reliable and valid.

That said, as a general best practice, you shouldn’t stop your studies early — even if  you see winning results. Doing so can provide inaccurate results that may lead to incorrect conclusions.


Analysis:

What does this study show us about the importance of Facebook ad testing? There’s a couple important takeaways:

1. Measure the “right” KPIs and look further down the funnel

This test shows the importance of looking at the right KPIs, further down the conversion funnel.

Had the team only looked at cost of the ad, and CTR on the creative, they would have incorrectly concluded the Halloween-themed creative was the winner.

But, because they went a step further and measured leads — the final KPI, at the end of the funnel — they were able to correctly conclude the Halloween-themed ad didn’t actually win because it lost on the most important metric, lead bookings.

This test shows the importance of measuring the right KPIs to truly determine which version wins.

2. Treats may look like tricks

In his influential book Pre-sausion, renowed author Robert Cialdini, describes how subtle visual cues can the set stage to prompt later conversions. These cues prime the user, opening them to become more receptive to the later the pitch, or ask.

In the Halloween-themed creative, the playful candy imagery set the tone for trick ‘or treating. The candy looked like a scrumptious treat. Facebook ad users may have been intrigued and went on to click the unusual creative. But, when confronted with the serious decision of needing to book a test drive, these users may have felt tricked.

The fun imagery in the playful Halloween ad perhaps didn’t appropriately set the stage to make such an important conversion decision. So, many users didn’t convert.

In contrast, the standard ad creative used the color blue — which is viewed as color of trust in many cultures — may have appropriately pre-sausively set the stage for a booking form users felt they could trust in divulging their personal contact information.


Tangible Takeaway & Immediate Application

Make sure to measure the right KPI, far down the funnel. Otherwise, you could incorrectly assume a winner where it doesn’t actually exist.

And, be mindful of the pre-sausive impact your imagery and ad creatives have on later conversion decisions.


What Do You Think?

Why do you think the standard ad creative generated more leads? Share your thoughts in the comment section below.

Poll Results - The Best Guesses:

Which Facebook Ad Generated More Leads?

  • Version A
  • Version B
Loading ... Loading ...

Sign-up for the newsletter to get FREE gamified A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test

5
COMMENTS: Tell Us Your Thoughts.

avatar
3 Comment threads
2 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
4 Comment authors
AdamChika ADeborah OMelissa Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Melissa
Member
Melissa

I chose version B because the creative looked more appealing and inviting, less distracting and cleaner. To me, the messaging was clearer in version B. But then again, who doesn’t love candy falling from the sky?

Chika A
Guest

Was the landing page also updated to reflect the theme of halloween to maintain consistency between the ad and the landing page?

Adam
Guest

Hi Chika, unfortunately, in this case, we weren’t able to edit the landing page. So both went to the dealers’ inventory page for the vehicle included in the offer.

Thanks!

Other Tests You Might Enjoy

Test Run By

Iacoca Blue Advertising

Test Run For

Hyundai

Test Run On

Facebook