Difference Between Versions:
Version A – Bottom notification bar, appearing when landing on the page, encouraging prospective students to learn more about the program of interest
Version B – Pop-up displaying when motioning to exit the page, encouraging prospective students to learn more about the program of interest
Key Performance Indicator (KPI):
Clickthrough Rate (CTR)
Get more prospective students clicking into program detail page
Prospective students interested in applying for programs
GMercyU - Private American university offering more than 40 degree programs
Sign-up for the newsletter to get FREE A/B tests and more valuable content delivered to you every other week.
Become a More Successful Digital MarketerPro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage
This week’s Featured Test is brought to you by:
Poll Results - The Best Guesses:
Which format converted best?
- Version B
- Version A
The exit pop-up won big-time!
But, the test was much more complex than simply comparing these two formats. Read on to learn more. . .
Test Details & Background:
NextLeft, an award-winning digital marketing agency, conducted this fantastic format test for their client GMercyU, a private American university, which offers over 40 undergraduate and graduate degree programs.
The test goal was to determine the optimal format for moving prospective students deeper into the website — from a general program overview page, called a Learn page, to a more detailed Program page, fully describing the academic curriculum.
The challenge was that each Learn page contained a lot of information, with many competing Calls To Action (CTAs).
Without markedly changing the page structure, or content, the testing team needed to find a simple solution to clearly direct prospective students on to the Program page.
It was initially suspected a notification bar could help draw attention, pointing users from the Learn to the Program page.
So, bottom notification bars were set-up across about 40 of the site’s Learn pages. The notification bar CTA buttons encouraged students to click into the related Program page.
The bottom notification bar displayed on both desktop and mobile devices. It looked like this:
While the notification bar seemed a simple solution to increase the Clickthrough Rate (CTR) from Learn to Program pages, it turned out to not be very effective.
With this data, the testing team went back to the drawing board.
They surmised the notification bar was likely too intrusive, taking up too much space — especially on small-screened mobile devices — about 50% of the site’s traffic.
So, decided to explore other formatting options that would help point prospective students in the right direction, from the Learn to Program pages.
The team devised three other formats that could be considered less intrusive:
- Bottom notification bar with no text, just CTA buttons
- Side hook, displayed on the lower, right side of the screen
- Pop-up that appeared when the user motioned to exit the screen
They decided to test these three variants against the bottom notification bar. The desktop views looked like this:
The mobile views looked like this:
Of all the variants, the testing team suspected the exit pop-up would outperform because it was the least intrusive and best-timed variant, leading to increased CTRs, from the Learn to Program pages.
They expected results to hold true across both desktop and mobile devices.
However, they also acknowledged that pop-ups are often viewed as annoying. And, over time, users have been conditioned to respond by immediately and instinctively closing pop-ups. As a result, the only interaction with the pop-up might be to close it.
With these factors in mind, they decided to test what would work best.
To determine the optimal notification format, an A/B test was set-up and run on the pop-up personalization platform, ConvertFlow.
The test ran across 8 Learn pages. These pages were specifically chosen because they had high traffic, encompassed a variety of popular program types, and could provide a good indication of broader user behavior, across all Learn pages.
To ensure adequate sample size, it was calculated at least 1,356 visitors would need to be see each variant.
To achieve this sample size, the test ran for 41 days, from April 1-May 12, 2020, which coincided with when COVID first hit.
Clickthrough Rates (CTRs) were tracked across the four different test variants.
A clickthrough conversion occurred when a visitor clicked on the CTA button and landed on the corresponding Program page.
The Real-Life Results:
Winner: Version B, the pop-up, was the strong winner, as hypothesized.
Compared to the control (bottom notification bar), the exit pop-up achieved an astounding 236.1% lift in CTR – increasing from 2.1% conversions to 8.1%!
Results achieved 95% confidence.
Further slicing the data, it’s interesting to note that results varied by device type.
As these charts show, overall, the pop-up outperformed because it achieved such a strong conversion lift on desktop.
But, on mobile, the hook actually outperformed.
However, looking at just mobile traffic, the sample size wasn’t large enough to declare statistically significant results.
Conversion rate by device type
As a result, the exit pop-up was declared the overall winner:
How Trustworthy Are The Results?
Given the sample size was large (nearly 20 thousand users), the test ran for a standard amount of time (41 days), and results achieved 95% confidence, it’s highly likely the study is valid.
However, a few important points should be noted:
- The exit pop-up ONLY showed to viewers motioning to leave the page/site. In contrast, all other notification bar formats were visible as soon as a the user landed on the page. Since the variable of when the notification displayed differed only for the pop-up, the test didn’t fully compare apples-to-apples from a notification bar timing standpoint. This timing difference could have impacted results.
- As well, because the exit pop-up triggered only when viewers motioned to leave the page, it received far fewer views than the other formats (approx. 2,900 views vs. 5,000 for each of the other variants). However, this sample size is still statistically significant. Interestingly, despite the smaller size sample, the pop-up converted better, both looking at total number of conversions and conversion percentage (or views/CTR).
- As luck would have it, this study started just as COVID hit. During some of the time the test ran, a bright, yellow COVID top notification banner displayed, globally on the site, in addition to the test notification bars. The dual notification bars could have impacted results. However, a subsequent test is now running — without the COVID notification banner. Preliminary results indicate the exit pop-up is continuing to outperform, showing user behavior during COVID, and the top COVID notification bar, likely didn’t markedly influence results of this study.
What do these results show us about notification bars and pop-ups? There’s a few insights that can be gleaned:
1. Pop-ups really do work!
In Conversion Rate Optimization (CRO) and A/B testing, there’s perhaps no strategy more scrutinized than applying a pop-up on your website.
For users, they can be seen as annoying, distracting, or intrusive.
But for marketers, they can help capture valuable information and lift conversion rates.
Because of this love-hate relationship, there’s a lot of debate and discussion about pop-ups. Marketers want to know two things:
- Should I use a pop-up on my site?
- Do they still really work? Or, are they now an old-school tactic and no longer convert?
The answer definitely seems to be: YES, you should use one, and, yes, they still work.
According to SUMO, the average conversion rate for a pop-up is 9.28%.
In addition to this case study, A/B tests like this one, this one, and this one, add further evidence to suggest that, despite increasing popularity, and more pervasive usage, pop-ups do indeed still seem to work!
2. Exit pop-ups might work best
However, there may be a caveat. Exit pop-ups might work best.
That’s because traditional, on-site pop-ups have become so prevalent, we’ve learned to adapt our browsing behavior. It’s now instinctual to close a pop-up as soon as it appears.
So, for a pop-up to now be most effective, it needs to be optimally timed.
This test indicates displaying upon exit may be optimal timing.
Showing a pop-up on exit gives visitors one last chance to take you up on your offer. And, this offer doesn’t compete with anything else on the page. It captures viewers’ full attention.
With full attention, users may have more desire to engage in the offer, and convert.
In this test, users came to the page, likely couldn’t find what they were looking for, and decided to leave. Upon exiting, they were prompted to revisit their decision with the information they may have been seeking in the first place. Their attention was caught, their interest sparked. So, decided to take action.
As an interesting follow-up to this test, a study is now being done looking at optimal notification bar timing. The exit pop-up is pitted against a bottom notification bar that displays when landing on the page, versus a notification bar that shows when the user scrolls down 20% of the page.
So far, the exit pop-up is outperforming, providing preliminary evidence that optimal timing is indeed upon exit.
But, of course, it’s always best to test what works best with your audience, on your website.
3. The more invasive the better!?
An interesting, and unexpected finding from this study was that, the larger and more visually prominent the notification format, the more likely it was to convert — across both desktop and mobile.
This outcome was surprising.
It was expected the larger the notification format, the more annoying, and therefore, the least likely to convert.
However, it seems that to capture full attention, invasive tactics are needed.
While a pop-up that appears upon exit isn’t overtly intrusive, it is invasive. When it takes over the screen simply can’t be ignored. The user needs to decide to either close it or convert.
But, if the pop-up contains anything of benefit, or value, the user might actually appreciate the in-your-face offer — and convert.
That said, Google continues to penalize interstitial pop-ups that take over the screen as soon as the user lands on the site. So, do use caution, and taste, when applying this powerful conversion technique.
Tangible Takeaway & Immediate Application
Exit pop-ups may be the optimal notification format to get user’s to take notice and convert.
What Do You Think?
Why do you think the exit pop-up won? What would you suggest testing next?
Share your thoughts in the comments section below.
Poll Results - The Best Guesses:
Which format converted best?
- Version B
- Version A