Difference Between Versions:
Version A – Halloween-themed creative; featured orange and black color scheme with candy background
Version B – Non Halloween-themed creative; featured blue and white color scheme with polka dots
Key Performance Indicator (KPI):
Average order value
Targeted Facebook ad users
Hyundai: Car company
Sign-up for the newsletter to get FREE A/B tests and more valuable content delivered to you every other week.
Become a More Successful Digital MarketerPro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage
This week’s Featured Test is brought to you by:
Poll Results - The Best Guesses:
The standard creative generated more leads (even though it lost on many other metrics)
Test Details & Background:
Iacoca Blue Advertising is a data-driven digital agency focused on creative advertising.
They conducted this tastefully treated Halloween Facebook ad creative test for their Canadian car client, Penticton Hyundai.
The test goal was to see if, during the month of October, the Halloween-themed creative would generate more leads, interested in booking a test drive.
With a successful A/B test result from a previous Halloween-themed Facebook ad experiment, the testing team wanted to verify if the holiday-themed creative would once again win.
Based on past experience, the agency expected the Halloween-themed ad would bring in more leads.
However, they also realized the spookily specific Halloween theme might not resonate that well with users throughout the entire month of October. So, decided to test what would work best.
To definitively determine which creative converted best, a Facebook ad A/B test was set-up and ran on Facebook, during the month of October.
The ad was broadly targeted.
The ad generated 23,362 impressions; traffic to the ad was split 50/50.
Half of viewers saw the colorful candy creative. It featured images of tempting treats in the background, with a Halloween-themed orange and black color scheme. The ad appeared like this:
The other half of Facebook ads users were shown the non Halloween-themed creative. It featured a fun polka dot background with a light and airy blue color scheme. It looked like this:
Leads clicking the “Learn More” button and filling out a booking form were tracked across both versions.
In addition, the team assessed the cost of the ads, both in terms of Cost Per Mille (CPM) and Cost Per Click (CPC).
The Real-Life Results:
Winner: Version B – contrary to the hypothesis, the non Halloween-themed creative won. But, not across all metrics.
—Cost Per Mille (CPM) and Cost Per Click (CPC)—
From a cost perspective, the Halloween-themed ad actually won. The Halloween-themed creative had a 6% lower CPM and 30% lower CPC rate, compared to the non Halloween-themed creative.
—Clickthrough rate (CTR)—
Looking at link clicks on the “Learn More” button, the Halloween-themed creative also had a 20% better CTR, compared to the standard creative.
But, despite the higher CTR, the halloween-themed creative actually generated 60% fewer leads than the regular ad. Now, that’s a scary stat!
It showed that, even though Facebook ad users were compelled to click on candy-colored creative, they weren’t ready to convert, or book a test drive.
Since the halloween-themed ad generated fewer leads — and the main objective was generating leads — the standard ad was declared the overall winner.
How Trustworthy Are The Results?
At the time of this publication, the final days of the study were wrapping up. Because the study hadn’t officially concluded within Facebook, a complete confidence level couldn’t be asserted.
However, given that the study ran for an adequate time period (nearly 1 month), and received a large number of impressions (23 thousand), it’s likely results hold reliable and valid.
That said, as a general best practice, you shouldn’t stop your studies early — even if you see winning results. Doing so can provide inaccurate results that may lead to incorrect conclusions.
What does this study show us about the importance of Facebook ad testing? There’s a couple important takeaways:
1. Measure the “right” KPIs and look further down the funnel
This test shows the importance of looking at the right KPIs, further down the conversion funnel.
Had the team only looked at cost of the ad, and CTR on the creative, they would have incorrectly concluded the Halloween-themed creative was the winner.
But, because they went a step further and measured leads — the final KPI, at the end of the funnel — they were able to correctly conclude the Halloween-themed ad didn’t actually win because it lost on the most important metric, lead bookings.
This test shows the importance of measuring the right KPIs to truly determine which version wins.
2. Treats may look like tricks
In his influential book Pre-sausion, renowed author Robert Cialdini, describes how subtle visual cues can the set stage to prompt later conversions. These cues prime the user, opening them to become more receptive to the later the pitch, or ask.
In the Halloween-themed creative, the playful candy imagery set the tone for trick ‘or treating. The candy looked like a scrumptious treat. Facebook ad users may have been intrigued and went on to click the unusual creative. But, when confronted with the serious decision of needing to book a test drive, these users may have felt tricked.
The fun imagery in the playful Halloween ad perhaps didn’t appropriately set the stage to make such an important conversion decision. So, many users didn’t convert.
In contrast, the standard ad creative used the color blue — which is viewed as color of trust in many cultures — may have appropriately pre-sausively set the stage for a booking form users felt they could trust in divulging their personal contact information.
Tangible Takeaway & Immediate Application
Make sure to measure the right KPI, far down the funnel. Otherwise, you could incorrectly assume a winner where it doesn’t actually exist.
And, be mindful of the pre-sausive impact your imagery and ad creatives have on later conversion decisions.
What Do You Think?
Why do you think the standard ad creative generated more leads? Share your thoughts in the comment section below.