Results: Which Numeric Format Worked Best?

Difference Between Versions:

Version A –Email opt-in offer stating 2 out of 3 individuals
Version B –Email opt-in offer stating 66% of individuals


Key Performance Indicator (KPI): 

Average order value


Test Goal:

Increase enrollment in the health insurance plan

 

Traffic Source:

Existing email list

 

Audience: 

Email list of subscribers who had opted-in for email updates, and created an account, but hadn’t yet started an insurance application

 

Organization: 

Centers for Medicare and Medicaid Services - the federal health agency that administers Medicare and Medicaid in the US

Sign-up to get FREE gamified A/B tests and more valuable content delivered to you each week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test
This week’s Featured Test is brought to you by:

Test Run By and For

Centers for Medicare and Medicaid Services

Test Run On

Google Tag Manager

WINNING VERSION

A

Poll Results - The Best Guesses:

Which Numeric Format Worked Best?

Loading ... Loading ...

Test Details & Background:

The Centers for Medicare and Medicaid Services, the federal health agency that administers Medicare and Medicaid in the US, ran this healthy study in-house.

The email team, within the Office of Communications, at the Centers for Medicare and Medicaid Services, wanted to know whether reporting information in “plain language,” using simple numbers would impact insurance applications.


Hypothesis:

They hypothesized using plain language — with smaller numbers – would be easier to understand. It would, therefore, be more compelling, leading to higher enrolment.

However, they acknowledged changing the presentation of the numbers might not have any affect. Or, could potentially diminish conversions.

So, decided it best to test.


 

Test Set-up:

To test, the effect of the numeric format, the team created two messages.

One was written with simple numbers. It stated 2 out of 3 individuals save on the insurance plans. It looked like this:

cms-lg-version_a

The same message was also turned more technical, by converting the figure into a percentage. It stated, 66% of individuals save on their insurance plans. The message looked like this:

cms-lg-version_a

The two versions were sent to an email list of people who had opted-in for email updates, created an account, but hadn’t yet started an insurance application.

To determine the optimal numeric format, an A/B test was set-up and run with Google, using Analytics and Tag Manager.

Form completions, leading to enrolment in a health insurance plan, were tracked across both versions.


The Real-Life Results:

Winner: Version A – the offer stating 2 out of 3 individuals was the definite winner.

Simply changing the number from 66%, to 2 out of 3, led to an astounding 106% increase in enrolment!


How Trustworthy Are The Results?

While interesting, these results can not be taken totally at face value.

Because the experiment was run within Google Analytics and Tag Manager, no confidence level was reported, or calculated. As well, neither the sample size tested, nor the length of time the experiment ran were reported.

Given these facts, the results may not hold true with other sample populations tested. So, if you’re thinking of taking this study as inspiration for your own work, you’re cautioned to test the effect with your audience before blindingly implementing — which, of course, is always best practice.


Analysis:

The presentation of numbers can heavily influence our perception. But, why did the smaller number win, in this case?

There’s likely a couple reasons:

1. We Don’t Fully Consider Variables

At a store, for example, an item priced at $99.99 appears less expensive than the same item going for $100.00.

In reality, the difference is only one cent. But, in our mind, the round number and extra zeros make the price seem a lot more.

This psychological phenomenon is referred to as numerosity, or the numerosity adaptation effect — which is discussed in-depth in this GuessTheTest article. According to some numerosity research, people tend to misjudge the quantity, or probability of numbers because they don’t fully consider variables, or unit sizes, impacting the number.

For example, a 12-ounce soda can might sell for $1.50. A similar 18-ounce can may go for $1.75. At first glance, both look similar in size, but the 12-ounce can seems cheaper.

However, breaking down the price on a per ounce basis, the larger can is actually the better deal. Yet, many people will gravitate towards the lower-priced can because they haven’t fully considered how unit size impacted price.

In this study, a similar phenomenon occurred.

People likely gravitated towards the smaller number because it was easier to access and visualize.

It was probably easier to imagine 2 out of 3 people saving. In contrast, it was harder to picture 66% of people; this percentage gave no reference to the total size of the group.

1. We Like To Visualize and Feel Part of a Group

Because the idea of 2 out of 3 people can be easily visualized, potential applicants could likely conjure images of being part of this small group.

Many readers probably felt the desire to become the missing third person. This want was likely further fostered through social proof.

In the next line of text, social proof was used to validate that ‘over 9.9 million’ others already signed up for the offer. The use of social proof, coupled with the easily understandable figure, likely helped drive conversions.


Tangible Takeaway & Immediate Application

When it comes to reporting numbers, this study shows presenting smaller, simpler figures may win. If you have the option to state a fact like 2 out of 3, or 66%, it may be beneficial to go with the smaller numeric format.

Mathematically, both break down to the same result. But, they may not be equal when it comes to conversions.


What Do You Think?

While numerosity, visualization, and social proof may have influenced the result, we certainly won’t count on it! 😉

Why do you think the smaller number won? Share your thoughts in the comments section below.

Poll Results - The Best Guesses:

Which Numeric Format Worked Best?

Loading ... Loading ...

Sign-up to get FREE gamified A/B tests and more valuable content delivered to you each week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test

2
COMMENTS: Tell Us Your Thoughts.

avatar
1 Comment threads
1 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
2 Comment authors
Deborah OWendy M Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Wendy M
Guest
Wendy Melemed

This is very interesting. I wonder if 2 out of 3 also sounds more impressive than 66%. 66% feels like just over half. 2 out of 3 feels more significant to me.

Other Tests You Might Enjoy

Test Run By and For

Centers for Medicare and Medicaid Services

Test Run On

Google Tag Manager