WINNING VERSION

A

Hypothesis:

The testing team suspected the low-friction variant "More information" would win since clicking to get more information seemed like an easy, low-commitment action to take.

However, they also acknowledged this phrase might come across as too gentle, and it wouldn't spur visitors into action.

So the team decided to test which wording worked best: low friction or high friction.

Test Set-up:

To definitively determine which format performed better: an A/B test was set-up and run on the programatic advertising platform, Xandr.

The study ran for 2 weeks and the ads received over 5 million impressions.

Traffic was split 50/50 across the 2 variants.

Version A: lower friction, "More information"

Half of shoppers (approximately 2.5 million) saw the banner ad with the low-friction copy.

On the left side of the banner, the was some white text, on a red background, advertising Internet, TV, and calling plans starting at €30.

A bold, bright, purple CTA button, stated the low-friction phrase, "More information".

On the right side of the banner, a young girl, wearing a frilly, pink dress was shown facing a TV, enthusiastically watching a rock concert. Her right arm was raised and her fingers communicated love in sign language 🤟❤️.

The banner ad looked like this:

Version B: higher-friction, "Order"

The other half of shoppers (approximately 2.5 million) saw the exact same banner ad, but with one key difference: the button copy changed.

In this version, the button copy contained a high-friction variant which stated, "Order".

It looked like this:

Clickthrough rate (CTR) on the banner was tracked across both versions.

The Real-Life Results:

Winner: Version A - the low-friction CTA button heightened conversions.

Using the phrase, "More information" increased CTR 6%, compared to the banner with the high-friction wording, "Order".

Results achieved a statistically significant effect at 95% confidence.

How Trustworthy Are The Results?

To determine test trustworthiness, a number of criteria needs to be evaluated, including testing timeframe, sample size, Sample Ratio Mismatch (SRM), and statistical significance.

Timeframe:

As outlined in this article, as a testing best practice, studies should, ideally, run for a period of 2-6 weeks. At 2 weeks, this study sat just within this ideal timeframe.

Sample Size:

As this article explains, for test results to be trustworthy, the study must include a large, representative sample of users.

Otherwise, it will not be adequately powered to accurately detect a significant effect, or conversion difference. In other words, results will be suspect.

Sample size calculations should always been done ahead of running the study using a sample size pre-test calculator like this one or this one. (For more on how to use these calculators, check out these GuessTheTest resources: calculating Minimum Detectable Effect (MDE), calculating power).

Unfortunately, the exact sample size data was not provided. We only know the ad received 5 million impressions. This amount was likely enough to run a properly-powered study.

Minimum Detectable Effect (MDE) & Power:

According to some stats experts, an MDE of 2-5% is considered reasonable.

An effect, or conversion difference, above 5% often gives reason to be suspicious. It indicates the study was likely not properly powered.

When tests are not properly powered, the conversion difference often becomes large and may be exaggerated, showing double or even triple-digit gains.

With a conversion lift of 6%, the effect size sits just above this reasonable upper bound, suggesting the study was likely properly powered.

Sample Ratio Mismatch (SRM):

As this article explains, SRM occurs whens test traffic is unexpectedly unevenly distributed to one variant.

This unequal weighting can taint results because traffic and conversions are potentially skewed, so you want to avoid SRM.

Unfortunately, there was not enough information reported with this study to know if SRM occurred.

Statistical Significance:

As explained in this article, statistical significance shows a test result didn't just occur out of random chance or error.

Rather a true winner has been detected. Confirming a test has achieved +95% statistical significance, based on a p-value of <0.05 at a power of +80% , is considered standard best practice.

Unfortunately, given the data provided, it's not possible to verify statistical significance.

Trustworthiness Summary:

While this test is interesting and appeared to achieve statistically significant results, the outcome can't be verified.

Why This Test Still Holds Merit:

Although not enough data was reported to verify the study results, the win was compelling enough for the testing team to implement the result.

Analysis:

Based on these findings, what does this test tell you that can be applied to optimize your own ads and CTA buttons?

There are a couple tangible takeaways:

Low-friction copy appears to perform better earlier in the funnel

This study adds to a growing body of research, including this test, this one, this one, this one, this one, and this one which all show, low-friction copy appears to perform better earlier in the funnel.

In contrast, this study and this one suggest higher-friction copy seems to work best only towards the bottom of the funnel -- after users have been exposed to your brand, and right before they have to make a final conversion decision.

Why does low-friction copy appear to work best early on?

Likely because it gently eases visitors into the funnel without too much pressure. Visitors at earlier stages tend to still be gathering information and are not quite ready to make a purchase decision.

So, when deciding what CTAs to use, evaluate where users are in the funnel, and cater to them accordingly.

** TEST IDEA ** -- Look your banner ads or top-of-funnel CTA copy.

Do you use low-friction phrases, like "see more" or "get more information" to grab visitors' attention, peak their interest, and gently nudge them into your sales funnel?

Or do you put the pressure on with heavy-hitting, commanding orders like "buy now" and "order today"?

Test the effect of using soft, low-commitment language to ease and encourage visitors into your funnel.

This language is likely to make users most comfortable early in the funnel, when they're still getting acquainted with your brand and offerings.

Much deeper in the funnel -- at the crucial decision point, where they'll choose to buy/sign-up -- test using more forceful, higher-friction language to push hesitant users down through the bottle neck of your funnel and into action.

See this diagram for a visual explanation of how this idea can work:

Connotation matters for conversions

All words have meaning. Connotation is the emotional association of words.

The connotation of words can affect our moods and subsequent behavior.

In fact, according to ecommerce research from the Missouri University of Science and Technology, emotional response strongly impacts visitors’ buying decisions.

According to some advertising psychology research, emotional response influences buying intention more than ad content -- even if we don't rationally think that's the case.

When we see high-friction words early in the funnel, it may create a negative emotional response. In contrast, the low-friction wording may cause a more positive emotional reaction because the user isn't being asked to commit to anything.

Research also shows positive messages, in the right context, are more effective; they're better at grabbing visitor attention and at getting people to remember your message.

Feeling more positive because the commitment level was low, visitors likely felt more comfortable clicking the CTA.

** TEST IDEA ** -- Assess the connotation of words you use with your copy and CTA buttons.

Early in the funnel, test the effect of using low-friction, action words like "get" "take" or "see".

These verbs all suggest the user can benefit without having to invest too much in the activity.

Tangible Takeaway:

The words we choose can have a big impact on users’ decision to act.

Test using low-friction, high-benefit words early on in the funnel.

The subtle connotation, or emotional association, of words can influence users’ behavior. So choose your words wisely.

What Are Your Thoughts?

What are your thoughts? Why do think the low-friction copy won?

Share your thoughts and comments in the section below.