Results: Did one version outperform?

Similar "Download Trial" CTA

Click here to enlarge

Did one version outperform?

Loading ... Loading ...

Difference Between Versions:

Version A – “Download Trial” CTA
Version B – Similar “Download Trial” CTA


Key Performance Indicator (KPI): 

Orders/visitor


Test Goal:

Increase clicks on the "Download Trial" button

 

Traffic Source:

Direct

 

Audience: 

Included all visitors from the U.S. and Canada

 

Organization: 

Autodesk - Makers of AutoCAD, the 3D CAD modeling software

Sign-up to get FREE gamified A/B tests and more valuable content delivered to you each week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test
This week’s Featured Test is brought to you by:

Test Run By and For

Autodesk

Test Run On

Adobe Target

WINNING VERSION

B

Poll Results - The Best Guesses:

Did one version outperform?

  • Version A
  • Version B
Loading ... Loading ...

Guess what!?! There is no difference between the variants.

So, if you spent a lot of time go back and forth, squinting, trying to determine what changed between the two versions, and couldn’t find anything, worry not. You don’t need new glasses. Phew. 🙂

But. . . And, this is an important but. . . even though there was no difference between the versions, this test still achieved a statically significant difference in results!

Version B won.

Perplexed?

Read on to find out more. . .


Test Details & Background:

Autodesk, the makers of AutoCAD, the popular 3D CAD modeling software conducted this A+ A/A test in-house.

An A/A test test takes a different testing approach. It pits two of the exact same versions against each other. A/A tests are performed to validate the test is set-up and working properly — and that the data coming back is clean, accurate, and reliable.

As a mature, and established, testing organization, AutoDesk decided to conduct 4 separate A/A tests, on different pages, with different amounts of traffic.

 

 


Test Set-up:

For all four A/A tests, the testing team set-up two separate pages that appeared exactly the same.

For this specific test, the pages looked like this:

autodesk-lg

With this test, over 17,000 visitors, from within North America, were directed to one of the two pages. Traffic was split 50/50. The experiment ran on Adobe Target for 17 days.

During this time, clicks on the “Download Trial” button were tracked across both versions.


Hypothesis:

The team expected there would be no difference in results between each version, for each test.

For 3 out of the 4 tests, this hypothesis was proven correct. No significant difference was found between variants.

However, for this test, the results were surprising. . .


The Real-Life Results:

Winner: Compared to experience A (the control), experience B achieved a 4.79% lift in orders/visitor. Results achieved 99% confidence.

Don’t believe it? You can see the results screenshot for yourself:

results_screenshot-autodesk


Analysis:

How can two versions of the exact same web page — with the exact same Call To Action (CTA) button — achieve markedly different results?

The only logical explanation is, it can’t.

There was likely an issue with the set-up, coding, or deployment of this particular test. Which goes to show, even a testing organization as experienced, and advanced, as AutoDesk is not immune to testing error.

And, that’s exactly why running A/A tests can be so valuable. No matter your experience, or proficiency, A/A tests can help you preemptively catch issues with your tests — before you implement the results.

If you get a significant A/A test result, it may show there’s a bug in the machine. Without running an A/A test, that bug might not get detected.

Ponder this idea. . .

Without A/A Testing, You Might Get False Positives

If results between the same pages, in an A/A test, can show a significant difference, imagine the wild results that can occur in an A/B test — with two different variants! The data can’t be trusted. But, to an unsuspecting marketer, the win may look huge.

In fact, it’s estimated that up to 80% of typical A/B test results are false positives. In other words, marketers think they’re getting significant test results, when they’re actually not.

This statistic is alarming considering most marketers implement apparent winning test results — and make key optimization decisions — based on data that might not be correct!

If you’ve ever run a test, achieved what you thought was a positive result, implemented it, and later realized the change wasn’t working, it’s likely you, too, have fallen victim to a false positive testing result.

As explained in this GuessTheTest article, A/A testing helps you rule out false positives, by indicating issues with the test itself — before you blindly implement the results.

AutoDesk’s Approach To Combating False Positives

According to Autodesk’s Senior Experience Researcher and UX Evangelist, Lisa Seaman, running A/A tests has given her team a “new appreciation for the power of larger traffic volumes, and the possibilities for false positives.”

To combat false positives, Autodesk recommends looking at the daily trend graphs within the testing platform.

According to Autodesk, even if the data on your A/B test shows you have a winner, if you see a lot of variability day-by-day, “you should take your results with a grain of salt. If you have a true winner, you should expect to see that experience winning on almost a daily basis.”

A/A Testing & Red Flags

That said, if you do achieve a statistically significant A/A test, it’s always prudent to make sure the result is not due to a user error, or instrumentation effect. An instrumentation effect occurs when the testing tool, or instrument is set-up wrong.

To ensure no instrumentation errors occur, it’s always valuable to make sure you:

  1. Cross-check the data with a secondary analytics tool. For example, integrate Google Analytics and set-up custom dimensions, in combination with your testing tool reports, to enable further analysis, beyond what’s reported in the testing software.
  2. Accurately set-up the test and code everything properly. Everything from small coding mistakes to errors that slow load time can cause a difference in results between the same versions. So, ensure everything looks in tip-top shape before you run any kind of test.
  3. Run the test long enough, and aim for 95%+ level of confidence. Stopping a test too early can impact results. You might think you have a winner when you don’t. Or vice versa. As well, as explained in this Optimizely blog, the higher the level of confidence, the less likely the result is to be due to chance. So, aim to achieve at least 95%+ confidence in your tests.

Tangible Takeaway & Immediate Application

There is utility in running an A/A test to ensure any results you obtain for future tests are accurate, and reliable.

Without running an A/A test, you risk implementing a “winning” test result that is actually a false positive.


What Do You Think?

Why do you think this A/A test achieved the result it did? And, what would be your suggested next step?

Share your thoughts in the comments section below.

Poll Results - The Best Guesses:

Did one version outperform?

  • Version A
  • Version B
Loading ... Loading ...

Sign-up to get FREE gamified A/B tests and more valuable content delivered to you each week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test

3
COMMENTS: Tell Us Your Thoughts.

avatar
2 Comment threads
1 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
2 Comment authors
Deborah OSteven Graff Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Steven Graff
Guest
Steven Graff

Often use A/A tests to confirm testing tool validity or test is set up correctly.

Other Tests You Might Enjoy

Test Run By and For

Autodesk

Test Run On

Adobe Target