Results: With or without the pre-selected amount?

With or without the pre-selected amount?

Loading ... Loading ...

Difference Between Versions:

Version A – Option to top up credit card balance using pre-set buttons
Version B – Option to top up credit card balance using fillable field


Key Performance Indicator (KPI): 

Top-up balance rate, top-up amount


Test Goal:

Determine the optimal format to get more customers topping up their credit card balance

 

Traffic Source:

All traffic

 

Audience: 

Turkish customers ordering food delivery

 

Organization: 

Yemeksepeti: Turkish online food delivery service

Sign-up for the newsletter to get FREE A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test
This week’s Featured Test is brought to you by:

Test Run By and For

Yemeksepeti

Test Run On

Optimizely

WINNING VERSION

A

Poll Results - The Best Guesses:

With or without the pre-selected amount?

  • Version A
  • Version B
Loading ... Loading ...

With the pre-set buttons won.

But only by default. Read on to learn more. . .


Test Details & Background:

Yemeksepeti.com, the Turkish online food delivery company, conducted this tasteful test in-house, on their site.

Through an online wallet system, Yemeksepeti enables users to maintain a credit balance, within their account, making it faster and easier to pay online, for upcoming orders.

When the balance runs low, customers are encouraged to top-up the amount.

Yemeksepeti’s testing team has found prompting users to maintain a high account balance leads to more food orders, and food orders of greater value.

So, they deemed it highly beneficial to test the optimal top-up forma to encourage customers to top-up their credit card balance.

Originally, the Yemeksepti site featured a top-up screen with three pre-defined amounts (25, 50, 100 TL) as well as a blank field where the user could input any amount desired. It looked like this:

 

version_a-with_preset_buttons

However, tThe testing team wanted to determine if removing the pre-defined amount buttons – replacing them with a bigger input box – would increase both the top-up rate as well as the top-up amount. So, they created a revised version which looked like this:

version_b-without_preset_buttons


Hypothesis:

The team suspected removing the pre-defined amounts work well since it would give users the freedom to enter the exact amount desired.

As a result, the top-up rate and amount would increase.

However, the team also acknowledged leaving the amount field blank slate could overwhelm visitors with choice, to the point they’d make no decision at all – and wouldn’t fill in the amount box.

So, decided to test which format worked best.


Test Set-up:

To definitively determine the optimal top-up format, an A/B test was set-up and run on Optimizely.

The test ran for 40 days. During this time, over 20 thousand visitors took part. Traffic was split 50/50.

Half of visitors saw the original version (control), with the pre-selected amount buttons. The other half of visitors were directed to the version with just the blank field.

The metrics of top-up rate and top-up amount were tracked across both versions.


The Real-Life Results:

Winner: Version A, the pre-filled buttons, won. But, only by default. . . 

Contrary to the hypothesis, the new design, with the blank field did not work better.

Top-up Rate

In fact, it led to a -11.18% drop in the top-up rate! Results achieved 90% confidence.

Top-up Amount

That said, the blank field design did have some merit. It worked better at increasing the average top-up amount. But, because far fewer users took the action of inputting their top-up amount, this blank slate version, overall, far under performed.

The team declared the test unsuccessful, but have decided to do future testing on the pre-selected amount buttons.


How Trustworthy Are The Results?

Given the sample size was large (20 thousand users), and the test ran for a standard amount of time (40 days), it’s highly likely the study is valid.

However, it should be noted, the results only achieved 90% confidence, which means there’s a 10% chance results would not hold true in future studies. That’s a wide margin. Ideally, you should aim to achieve 95%+ significance in your studies.

As well, it should be noted, the top-up amount was not reported. Therefore, we have to take the testing team’s word for it when they say the losing variant overall underperformed.


Analysis:

What does this surprising study show us? There’s a couple important takeaways:

1. Amount of choice impacts conversions

As previously discussed, choice can impact conversions.

Too little choice, people feel disempowered and don’t act. Too much choice, people become overwhelmed with possibilities – and don’t convert.

On the version with the blank box, customers were faced with a lot of choices.

First, they had to decide whether or not they wanted to top-up their wallet balance.

And, if so, they then had to figure how much to add.

This choice was likely difficult for many hungry visitors who simply wanted to place their order and fill their bellies – not their online wallet.

Because there was no suggested figure to guide them, many users chose the easiest option: to not top-up at all.

More choice did not equal more conversions.

In contrast, the version that limited choice to a selection of clickable buttons helped guide hungry visitors. Interestingly, however, it also limited them; these users were less likely to top-up a high amount.

This test shows, when making a choice on whether or not to add money, most users need guidance. But, to truly optimize conversions the pre-determined amounts need to be carefully worked out so visitors give most.

2. Results are context specific and may differ – so test!

It’s interesting to note, not all the studies have arrived at the same finding.

In fact, two not-for-profit donation studies show markedly different results. In this test by the World Wildlife Fund, the pre-filled button amount was definitely the way to go. Just like this Yemeksepeti study.

However, in this donation test by Caring Bridge, the blank button format worked far more effectively.

Why the discrepancy across case studies?

There’s likely a couple factors at play:

  1. The context of the contribution – In this Yemeksepeti study, anyone adding funds was putting money into their own account. They weren’t giving or donating to others. This simple fact may have dramatically changed user behavior. Therefore, it’s hard to draw trends across case studies that are in different industry verticals and have far different contexts.
  2. Type of visitor – Additionally, anyone making a monetary contribution may either be doing so for the first-time or a repeat customer. The type of visitor may influence contribution behavior. A first-time user may not feel as comfortable forking over money as a person who’s established a trusting relationship with the brand.

These factors will influence how users behave, thereby affecting overall conversions.

Taken together, these studies show there’s not a right way to optimally format the contribution field on your site. The only right thing to do is test for yourself! 😉


Tangible Takeaway & Immediate Application

The amount of choice given, the context in which you’re asking for a monetary contribution, and the type of visitor (new versus returning) are some aspects that may determine optimal funding format.

Consider these factors when designing and testing your site’s optimal format.


What Do You Think?

Why do you think the version with the fillable field underperformed? What would suggest testing next?

Share your thoughts in the comments section below.

Poll Results - The Best Guesses:

With or without the pre-selected amount?

  • Version A
  • Version B
Loading ... Loading ...

Sign-up for the newsletter to get FREE gamified A/B tests and more valuable content delivered to you every other week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test
Subscribe
Notify of
guest
3 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Don Tepper
Don Tepper
1 month ago
Reply to  Deborah O

I’d always heard and read findings that presenting choices–the three buttons in this case–would improve response over an open-ended option. So, generally, no, I’m not surprised. One element not discussed above, and that is very important, are the choices being presented. In this case, it was 25TL, 50 TL, and 100 TL. What if, hypothetically, the test had been 30TL, 75 TL, and 150TL? Speaking personally, I’m often “turned off” when the amounts are set too high. This is especially true with charitable donations. I may initially be thinking of contributing $10, but then see that the choices are $50,… Read more »

Other Tests You Might Enjoy

Test Run By and For

Yemeksepeti

Test Run On

Optimizely