Summary:

The McClatchy Company, a leading print and digital news provider, conducted this “newsworthy” study in-house, using Adobe Target.

Testing the most effective navigational menu (nav menu) copy, the team found the wording “Classifieds” catapulted conversions.

Compared to the “Buy & Sell” title, the winning “Classifieds” wording increased clicks on this section of the navigation bar by an incredible 75%!

Results achieved 99.9% confidence.

Test Details:

For media mogul, The McClatchy Company, immediately capturing readers is of prime importance. Through rigorous testing, the team has learned the optimal navigational menu (nav menu) format can make a big difference with reader conversions.

In a previous study, McClatchy examined whether a “kebab” menu format outperformed a “cog” or “avatar” format on mobile. Similar to this Intel test, McClatchy found the format shown greatly impacted conversions. In this case, the control “kebab” format was the big winner, increasing the Clickthrough rate (CTR) 10.9%, compared to the other variants.

Turning their attention to the menu copy, the testing team also wanted to determine if there was optimal wording that would appeal to their visitors.

Through Quancast analytics data, the media company learned the majority of its readers were males over the age of 35, with children. This readership base also tended to be highly educated and affluent.

Given its younger, forward-thinking audience, McClatchy’s testing team wondered if the original “Classifieds” menu title sounded outdated to its hip audience; they questioned if the fresher title “Buy & Sell” might convert better.

Background:

To test, the team took a two-pronged approach to determine the best navigational menu wording.

They began with qualitative User eXperience (UX) testing, observing user behavior and comments about the menu wording.

Indeed, UX results revealed people were able to more easily navigate when the title was switched to “Buy & Sell.”

So it was changed.

However, both the UX and analytics teams wondered if the results were completely accurate. Across teams, everyone agreed, users sometimes say one thing, but act another way.

So, quantitative A/B testing was done to verify the UX study results.

Hypothesis:

The analytics team hypothesized, as a newspaper site, the original “Classifieds” label would work better than the newer “Buy & Sell” title.

However, not wanting to discount the excellent work of the UX team, they figured it’d be best to test.

Test Set-up:

To determine which navigational menu title worked best, traffic was split 50/50.  Over 417,500 visitors saw either the navigational title with the updated title “Buy & Sell,” which looked like this:

Or the classic “Classifieds” title, which looked like this:

McClatchy’s testing team ran the study in-house, on their site. The test lasted for 26 days and was run using Adobe Target.

The Real-Life Results:

Contrary to what UX testing revealed, A/B test results showed “Classifieds” was the clear winner!

Simply changing the copy from “Buy & Sell” to “Classifieds” catapulted menu clicks 75%, at 99% confidence.

However, it’s interesting to note, the gains were on desktop only.  Although the study ran on all devices, no other results achieved statistical significance.

It’s also important to note that, although McClatchy typically tests deeper in the funnel — beyond just clicks — with this test, volume was too low to confidently assert statistical significance beyond top-level navigational menu clicks.

Analysis:

What people say they want and what they actually prefer can be different.

As such, a qualitative user experience focus group might reveal one finding that completely contradicts quantitative, hard evidence.

As UX expert Gerry McGovern puts it in his article Why focus groups don’t work, “the biggest problem in getting to know your customers is that they don’t know themselves.”

In one UX study, McGovern reports an airline whose passengers said they wanted healthy food onboard — not chocolate and candy. But, when the airline changed it’s menu to offer primarily healthy options, their fruits and vegetables went rotten. In reality, people really wanted the junk food — even though they said otherwise.

Because people may say one thing, but act differently, it can be problematic to completely trust the results of qualitative UX studies. Anecdotally, I’ve conducted several UX studies where I’ve watched participants struggle to achieve the top task and locate the information they’ve been asked to find on the website. But, at the end of the study, when asked their thoughts on the site, they say, “it’s a great website; it’s easy to use and find all the information.”

Because there’s a bias that can occur when people relate to people in UX studies, the most robust way to get hard data is to confirm results through quantitative A/B testing.

Additionally, this test shows it’s crucial to really get to know your audience — beyond what they say they want — by learning how to best speak to them in terms they’ll understand.

Although “Buy & Sell” may have seemed a more intuitive and informative label, McClatchy’s core audience was used to the traditional “Classifieds” newspaper label. They, likely, instinctively looked for it to buy or sell something.

Because they were most accustomed to seeing the word “Classifieds,” it converted best in a real-life situation.

This test shows, you can’t always trust your gut — or the advice of your users.

Even if popular belief dictates a certain viewpoint, always push to A/B test. Get hard evidence. Then make an informed decision, based on the combination of both qualitative and quantitative data.

This test did an excellent job of synthesizing qualitative and quantitative data to arrive at a great, informative result!

Actionable Takeaways:

1. What People Say The Want — And How They Really Act — Can Be Different!

Don’t offer fruits and vegetables when your customers really want chocolate — even if they won’t admit it. Go beyond what your customers say they want and rigorously test to find out what they really want. Then, speak to them in wording they’ll best relate to and understand.

2. Combine Both UX Testing and A/B Testing For The Most Robust Results

As this study shows, qualitative test results don’t always match up with quantitative results — which brings up the question, what results can you truly trust? The best way to arrive at a valid conclusion is to conduct both UX and A/B tests, rigorously analyze the data, then work to understand your audience, their needs, and where they may be saying one thing, but acting differently. Based, on this in-depth analysis, you’ll be further ahead at implementing a solution that truly works best for your audience.

What Are Your Thoughts?

Why do you think the wording “Classifieds” converted best? And, how do you run valid UX and A/B studies? Share your thoughts in the comments section below.

What was it about the wording that worked so well? And, how can you apply these principles to your nav menu?
Unlock Pro Membership Access