Results: Does segmentation really work? Or, does it backfire?

Generic page; not segmented by user's industry vertical

Click here to enlarge

Dynamically changing page; segmented by user's industry vertical

Click here to enlarge

Does segmentation really work? Or, does it backfire?

Loading ... Loading ...

Difference Between Versions:

Version A – Generic homepage and landing page image, copy, and CTA. Targeted for the general cloud computing sector
Version B – Dynamically changing homepage and landing page image, copy, and CTA. Segmented by industry vertical; this version was targeted to visitors within the banking sector


Key Performance Indicator (KPI): 

Clickthrough Rate (CTR) and Download Rate


Test Goal:

Increase CTR to the landing page, and increase product downloads

 

Traffic Source:

Direct and organic traffic

 

Audience: 

Visitors to the German SAP.com homepage; visitors were from the banking, retail, higher education, automotive, or consumer goods industries

 

Organization: 

SAP Germany: The German division of the global technology solutions provider

Sign-up for the newsletter to get FREE A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test
This week’s Featured Test is brought to you by:

Test Run By

SAP

Test Run For

SAP

Test Run On

Adobe Target

WINNING VERSION

B

Poll Results - The Best Guesses:

Does segmentation really work? Or, does it backfire?

  • Version B
  • Version A
Loading ... Loading ...

Yes, it really does work!


Test Details & Background:

SAP, the multi-national enterprise application software provider conducted this dynamic test in-house, on the German division of their site.

With a diverse global clientele, one of SAP’s challenges is creating relevant customer experiences, site-wide. What’s added to the challenge is that 90% of SAP’s visitors disable cookie tracking, meaning they remain anonymous when on the SAP site.

Without a lot of customer information, it’s been difficult for SAP to create tailored web experiences.

But, realizing the power of personalization, the testing team wanted to know if creating dynamically changing homepage and landing page designs — segmented by industry vertical — would be worth the effort, and lift conversion rates.

To personalize the web experience by industry vertical, the team tested dynamically changing three elements on the homepage and landing page:

  1. The main image
  2. The central body copy
  3. The key Call To Action (CTA)

Visitors were segmented by industry. Five sectors were tested. They included:

  1. Banking
  2. Retail
  3. Higher education
  4. Automotive
  5. Consumer goods

 

Hypothesis:

The team suspected a segmented industry experience would resonate better than the generic site, resulting in a higher clickthrough rates and product download rates.

However, they acknowledged that the effort required in creating a dynamically changing site, segmented by industry vertical, could not be worth it. Or, could backfire, if the vertical didn’t present properly to an anonymous visitor.

So, decided to test what worked best.


Test Set-up:

To determine which experience would win, the testing team set-up and ran an A/B test on Adobe Target, with insights gleaned from Adobe Analytics.

The test ran for three months. During this time, 100 thousand visitors took part. Traffic was split evenly across the five industry verticals.

As a result, 20 thousand visitors were exposed to each of the 5 industry verticals. Within this subset, traffic was split 50/50. Half of users were directed to the generic page. The other half of visitors saw the dynamically personalized page, based on their industry vertical.

The generic page design featured an image a climber standing proudly on top of a mountain peak, surrounded by clouds. The website copy and CTA were broadly targeted, and meant to be generic for anyone in the cloud computing sector. The homepage and subsequent landing page design looked like this:

generic_experience

As an example, this design featured one of the industry verticals: the higher education sector. The experience showed engaged students in a classroom environment. The website copy and CTA were targeted to resonate with readers within the education sector. The homepage and subsequent landing page appeared like this:

higher_education

The banking experience featured a glowing skyline. The website copy and CTA were targeted for audiences within the banking industry. The homepage and subsequent landing page looked like this:

version_b-lg

Clickthrough rates (CTRs) from homepage to landing page, and product downloads were tracked across all versions.


The Real-Life Results:

Winner: Version B – without question, the personalized pages popped.

It’s perhaps not surprising the dynamically changing page won. But, what is surprising is how much customer segmentation impacted conversions, especially within each industry vertical.

—Clickthrough rate (CTR)–

Aggregating the results of the segmented experiences — against the generic control — the personalized pages lifted CTR 59%.

But, some industries performed better than others.

The banking sector was standout with a CTR lift of +269%, compared to the control!

–Download rate–

Overall, across all industries, the download rate for the segmented pages increased 28.96%, compared to the control.

For the banking experience, the download rate lift was +195.77%, compared to the control. Talk about a vertical increase!

Results achieved 95% confidence.


How Trustworthy Are The Results?

Given that the sample size was large (20,000 visitors per industry segment), the test was run for an extended time period (3 months), and results achieved 95% confidence, the test appears to hold strong validity.

However, it’s unclear how individual users were segmented, if their cookies were turned off and their data anonymous.


Analysis:

Why did the personalized design win? There’s likely a couple reasons:

1. Personalizing is a powerful conversion booster

Research looking at consumer response to personalization has found groups that share common traits, like industry verticals, tend to respond best to personalized content they can most relate to.

Perhaps why this reaction occurs lies in social proof  a psychological phenomenon where individuals look to others to dictate acceptable behavior.

2. Segmentation works well 

From the lens of social proof, it can be reasoned that when an individual is segmented to — and made to feel part of the collective — they become more engaged in the content. The experience feels more relevant.

Drawing on the idea of relevance, SAP is continuing to run more industry segmentation tests.

And, they’ve arrived at an unexpected finding. . .

The team has found segmentation works best in industries where workers can easily identify with all faucets of the profession. For example, a person who works in the banking sector can more easily identify messaging surrounding the industry.

In contrast, someone from the retail industry may not have the same connection because they may instead identify more closely with the supply chain, or fast-moving consumer goods segment, rather than a different industry sub-sector.

Focussing on the “easy to identify” industries has now become SAP’s goal.

The team is now globally rolling out a “segmentation by industry” campaign. Their findings so far are clear: Segmentation unquestionably increases CTRs, engagement, and download rates – no matter the culture, or where people are in the world.


Tangible Takeaway & Immediate Application

Pages targeted to specific audience segments tends to perform far better than content generically developed for everyone.

That’s because different audiences behave differently. A one-sized fits all solution really fits no one.

To optimize conversions, it’s important you learn as much as you can about your audience. Know them intimately. Find out what keeps them awake at night. Then target content to them appropriately.


What Do You Think?

Why do you think the personalized pages won? Share your thoughts in the comment section below.

Poll Results - The Best Guesses:

Does segmentation really work? Or, does it backfire?

  • Version B
  • Version A
Loading ... Loading ...

Sign-up for the newsletter to get FREE gamified A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test

5
COMMENTS: Tell Us Your Thoughts.

avatar
2 Comment threads
3 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
3 Comment authors
Deborah OAlexWendy M Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Alex
Guest
Alex

HI, I am just a bit puzzled about the fact the 90% of their user base don’t accept the cookies, that means there may be some question marks about the results (as mentioned above in the conclusion). But maybe they do install tracking code even if the user doesn’t agree with all cookie settings. So I guess they could build some segments and trigger page variants based on those ones.
Anyway I think test they have run it is very interesting and I am happy to see that they got significant results.

Other Tests You Might Enjoy

Test Run By

SAP

Test Run For

SAP

Test Run On

Adobe Target