Results: Which landing page format won in December, but lost in January?

Longer landing page copy

Click here to enlarge

Shorter landing page copy

Click here to enlarge

Which landing page format won in December, but lost in January?

Loading ... Loading ...

Difference Between Versions:

Version A – Landing page for a magazine subscription, with longer copy
Version B – Similar landing page, but with shorter copy


Key Performance Indicator (KPI): 

Add to cart conversions


Test Goal:

Assess if seasonality affected the effectiveness of a landing page for a magazine gift subscription

 

Traffic Source:

Visitors from Manchester United Football Club's website

 

Audience: 

Visitors coming from the Manchester United website

 

Organization: 

Haymarket Publishing - privately owned online and print media company

Sign-up for the newsletter to get FREE A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test
This week’s Featured Test is brought to you by:

Test Run By

House of Kaizen

Test Run For

Haymarket Publishing

Test Run On

Visual Website Optimizer (VWO)

WINNING VERSION

B

Poll Results - The Best Guesses:

Which landing page format won in December, but lost in January?

  • Version A
  • Version B
Loading ... Loading ...

Interestingly, the shorter landing page copy won in December, but lost in January.

Read on to learn more about the effect of seasonality in testing. . .


Test Details & Background:

Haymarket Publishing UK is a privately-owned media company, with dozens of print and online publications, across a variety of industry verticals.

Working with the digital optimization agency, House of Kaizen, the team wanted to determine if the effectiveness of a landing page offer was affected by the time of year.

The landing page was crafted for a magazine subscription to Inside United, the official magazine of the Manchester United football team.

The value proposition within the landing page was tailored to resonate with Manchester United sports fans.

As the team observed browsing behavior, they suspected a different audience was landing on the page before Christmas – shopping for holidays gifts others. But, after Christmas, visitors were likely looking to purchase the subscription for themselves.


 

Hypothesis:

The testing team surmised a different landing page format would likely be more effective for the different user segments: gift shoppers or Manchester fans.

Gift shoppers, they figured, would likely respond best to a more direct landing page. Therefore, a shorter landing page format would perform best over the holiday period.

In contrast, once the holidays were over, users were likely looking to purchase the magazine subscription for themselves. As avid Manchester fans, shoppers would want more information about the magazine. A longer landing page would, therefore, work best.

However, the team also considered the opposite effect could hold true. So, decided to put the landing page design to the test.


Test Set-up:

To test the effect of changing the landing page, two different offers were created and compared.

Performance of the two landing pages was assessed both during the Christmas holiday season, and after the holidays, in January.

During this time, traffic was split 50/50.

Visitors either saw the longer landing page, which looked like this:

manchester-version_a-lg
Or, the shorter, more direct landing page offer, which looked like this:

manchester-version_b-lg

Add to cart clicks were tracked on both landing pages, across the entire time period of the study.


The Real-Life Results:

Winner: Version B – during the holiday season, the landing page offer with the shorter copy was the standout winner.

In the run up to Christmas, the shorter version saw a 14.29% increase in add to cart clicks, compared to the longer version.

However, in the post-Christmas period, the longer format actually won, increasing add to cart conversions by a strong 25.74%.


How Trustworthy Are The Results?

Although the study presents a very interesting perspective, examining the effect of seasonality with different audience segments, results must be taken with a grain of salt.

Unfortunately, no sample size or confidence level were reported for this test. As such, the study only anecdotally suggests seasonal factors impacted users’ purchase behavior. Additional testing is needed to confirm the validity of this assumption.


Analysis:

Why did one landing page perform better than another during different months? There are likely a couple reasons:

1. Seasonality can impact who is purchasing on the site

Different audiences have different needs and different mindsets.

During the holidays visitors might, for example, be primarily paid traffic users who are urgently looking for a gift, but would never visit the site otherwise.

These visitors will behave much differently than regular clientele who organically find the site.

As such, it’s important to assess whether visitor segments to a site differ during certain times of year, like the holidays.

In this study, most web visitors coming to the subscription site during the holiday season were probably not the product consumer; they were buying a gift for someone else.

Because they weren’t heavily invested in the purchase, they didn’t need to navigate through longer copy to become persuaded. Rather, they likely just wanted to make the purchase, and get on with their busy holiday shopping list.

After Christmas, however, everything changed.

Visitors landing on the page were no longer primarily buying for others. They were the end-users. The longer, more detailed copy provided value for fans who needed that extra nudge to decide whether to spring for the magazine subscription.

This study shows, running a test without comparing different audiences, across different time periods, could lead to conversion losses.

2. Seasonality impacts testing timeframes

Seasonality isn’t just about the holidays, however.

Depending on the type of business, you may find different consumer segments, from different locations, visit your site during different time periods.

For example, if you selling boating products on site, your audience may primarily be from warm, sunny Southern climates during the winter, but expand to more Northern locations in the summer.

If your audience changes over the seasons, you’ll want to consider demographic and geolocation factors over the entire year, and across the sample population. Failing to do so could greatly skew your test results, and impact of your sales the remainder of the year.

3. Timeframe also impacts sales cycles

As well, when testing, it’s important to take into account purchase patterns and buying cycles.

If you run a credit financing site, visits may spike in the period leading up to the holidays, but drop markedly post-holidays. Since you’ll want a representative sample of how all users behave, you’ll want to take this timing into account when planning your testing timeframe.

Additionally, if your sales cycle is two months, for example, but you’re only testing for two weeks, you won’t get a full and accurate picture of how your products are performing. In this case, you might want to extend your testing timeframe to cover your full sales cycle.

All in all, the best tests are ones that win — and results hold true all year long!


Tangible Takeaway & Immediate Application

Seasonal factors can influence who your users are, where they come from, and why they’re buying.

Incorporating seasonal factors into your testing timeframe can help you ensure your testing gains hold true – across the entire year.


What Do You Think?

Why do you think the landing with the shorter copy won in the lead-up to the holidays, but later lost?

Share your thoughts in the comments section below.

Poll Results - The Best Guesses:

Which landing page format won in December, but lost in January?

  • Version A
  • Version B
Loading ... Loading ...

Sign-up for the newsletter to get FREE gamified A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test

3
COMMENTS: Tell Us Your Thoughts.

avatar
2 Comment threads
1 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
2 Comment authors
Deborah OOwen Jones Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Owen Jones
Guest

I guessed correctly Deborah. Only by applying my own panic-like buying patterns when buying gifts and comparing it to a more relaxed, voucher-laden January Owen.

Other Tests You Might Enjoy

Test Run By

House of Kaizen

Test Run For

Haymarket Publishing

Test Run On

Visual Website Optimizer (VWO)