Yes, it really does work!

Test Details & Background:

SAP, the multi-national enterprise application software provider conducted this dynamic test in-house, on the German division of their site.

With a diverse global clientele, one of SAP’s challenges is creating relevant customer experiences, site-wide. What’s added to the challenge is that 90% of SAP’s visitors disable cookie tracking, meaning they remain anonymous when on the SAP site.

Without a lot of customer information, it’s been difficult for SAP to create tailored web experiences.

But, realizing the power of personalization, the testing team wanted to know if creating dynamically changing homepage and landing page designs — segmented by industry vertical — would be worth the effort, and lift conversion rates.

To personalize the web experience by industry vertical, the team tested dynamically changing three elements on the homepage and landing page:

1. The main image
2. The central body copy
3. The key Call To Action (CTA)

Visitors were segmented by industry. Five sectors were tested. They included:

1. Banking
2. Retail
3. Higher education
4. Automotive
5. Consumer goods

Hypothesis:

The team suspected a segmented industry experience would resonate better than the generic site, resulting in a higher clickthrough rates and product download rates.

However, they acknowledged that the effort required in creating a dynamically changing site, segmented by industry vertical, could not be worth it. Or, could backfire, if the vertical didn’t present properly to an anonymous visitor.

So, decided to test what worked best.

Test Set-up:

To determine which experience would win, the testing team set-up and ran an A/B test on Adobe Target, with insights gleaned from Adobe Analytics.

The test ran for three months. During this time, 100 thousand visitors took part. Traffic was split evenly across the five industry verticals.

As a result, 20 thousand visitors were exposed to each of the 5 industry verticals. Within this subset, traffic was split 50/50. Half of users were directed to the generic page. The other half of visitors saw the dynamically personalized page, based on their industry vertical.

The generic page design featured an image a climber standing proudly on top of a mountain peak, surrounded by clouds. The website copy and CTA were broadly targeted, and meant to be generic for anyone in the cloud computing sector. The homepage and subsequent landing page design looked like this:

As an example, this design featured one of the industry verticals: the higher education sector. The experience showed engaged students in a classroom environment. The website copy and CTA were targeted to resonate with readers within the education sector. The homepage and subsequent landing page appeared like this:

The banking experience featured a glowing skyline. The website copy and CTA were targeted for audiences within the banking industry. The homepage and subsequent landing page looked like this:

Clickthrough rates (CTRs) from homepage to landing page, and product downloads were tracked across all versions.

The Real-Life Results:

Winner: Version B – without question, the personalized pages popped.

It’s perhaps not surprising the dynamically changing page won. But, what is surprising is how much customer segmentation impacted conversions, especially within each industry vertical.

—Clickthrough rate (CTR)–

Aggregating the results of the segmented experiences — against the generic control — the personalized pages lifted CTR 59%.

But, some industries performed better than others.

The banking sector was standout with a CTR lift of +269%, compared to the control!

–Download rate–

Overall, across all industries, the download rate for the segmented pages increased 28.96%, compared to the control.

For the banking experience, the download rate lift was +195.77%, compared to the control. Talk about a vertical increase!

Results achieved 95% confidence.

How Trustworthy Are The Results?

Given that the sample size was large (20,000 visitors per industry segment), the test was run for an extended time period (3 months), and results achieved 95% confidence, the test appears to hold strong validity.

However, it’s unclear how individual users were segmented, if their cookies were turned off and their data anonymous.

Analysis:

Why did the personalized design win? There’s likely a couple reasons:

1. Personalizing is a powerful conversion booster

Research looking at consumer response to personalization has found groups that share common traits, like industry verticals, tend to respond best to personalized content they can most relate to.

Perhaps why this reaction occurs lies in social proof  a psychological phenomenon where individuals look to others to dictate acceptable behavior.

2. Segmentation works well 

From the lens of social proof, it can be reasoned that when an individual is segmented to — and made to feel part of the collective — they become more engaged in the content. The experience feels more relevant.

Drawing on the idea of relevance, SAP is continuing to run more industry segmentation tests.

And, they’ve arrived at an unexpected finding. . .

The team has found segmentation works best in industries where workers can easily identify with all faucets of the profession. For example, a person who works in the banking sector can more easily identify messaging surrounding the industry.

In contrast, someone from the retail industry may not have the same connection because they may instead identify more closely with the supply chain, or fast-moving consumer goods segment, rather than a different industry sub-sector.

Focussing on the “easy to identify” industries has now become SAP’s goal.

The team is now globally rolling out a “segmentation by industry” campaign. Their findings so far are clear: Segmentation unquestionably increases CTRs, engagement, and download rates – no matter the culture, or where people are in the world.

Tangible Takeaway & Immediate Application

Pages targeted to specific audience segments tends to perform far better than content generically developed for everyone.

That’s because different audiences behave differently. A one-sized fits all solution really fits no one.

To optimize conversions, it’s important you learn as much as you can about your audience. Know them intimately. Find out what keeps them awake at night. Then target content to them appropriately.

What Do You Think?

Why do you think the personalized pages won? Share your thoughts in the comment section below.