Results: Email or push notifications?

Email or push notifications?

Loading ... Loading ...

Difference Between Versions:

Version A – Email update sent to subscribers’ inbox
Version B – Push notification sent to subscribers’ desktop


Key Performance Indicator (KPI): 

Clickthrough rate (CTR) and subscriber list growth rate


Test Goal:

Increase newsletter subscriptions

 

Traffic Source: 

Existing email list and push notification subscriber list

Sign-up to get FREE gamified A/B tests and more valuable content delivered to you each week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test
This week’s Featured Test is brought to you by:

Test Run By

PushCrew

Test Run For

Visual Website Optimizer (VWO)

Test Run On

HubSpot

WINNING VERSION

B

Poll Results - The Best Guesses:

Email or push notifications?

  • Version B
  • Version A
Loading ... Loading ...

Background & Test Details:

Pushcrew is a push notification platform that lets you send interactive, clickable updates to your subscribers, leading them directly to your website.

Pushing the boundaries, Pushcrew’s testing team ran this notable test for the blog section of their sister site Visual Website Optimizer (VWO), the A/B testing platform. VWO’s blog covers various Conversion Rate Optimization (CRO) and testing topics.

The test goal was to determine if more readers would engage with new blog content when sent updates through email or push notifications.


Hypothesis:

The team suspected sending push notifications would increase reader engagement, since the notification system would facilitate a quicker, easier sign-up process.

Specifically, the testing team expected:

  • Clickthrough rates of promoted blog posts would increase through website push notifications, compared to email notifications.
  • Subscriber list rates would grow faster using push notifications, than through email sign-up.

 

Test Set-up:

To determine the effectiveness of browser-based push notifications, the PushCrew team assessed sign-up rates across 13 different blog posts, published on the VWO blog site.

Readers were either sent an email, push notification, or both. The email template was formatted to look like this:

version_a-lg

The same 13 blog posts were then sent, to push notification subscribers. The push notifications looked like this:

version_b-lg

There was a strong overlap between the email and push notification distribution list. If users opted into both email and push notifications, they received both messages. However, push notifications were sent a week later than the email, to reduce reader fatigue.

This content distribution process continued for 4 months. During this time, over 35,000 subscribers received either emails and/or push notifications.

Variations in clickthrough and subscriber list rates were tracked between the two communication channels, using HubSpot.


The Real-Life Results:

Winner: Version B – as the team expected, push notifications pushed conversions much higher.

With push notifications, clickthrough rates increased an astounding 331%, compared to email!

And, the push notification list grew 8 times more than the email list.

However, once visitors clicked to the content, there was no difference in the amount of time spent on the page. This results shows the notification system only impacted the likelihood of whether readers would click to the content – not how engaged they were once reading.


How Trustworthy Are The Results?

This outcome seems moderately trustworthy, based on the large sample size of 35,000 subscribers. However, it should be noted, no confidence level was reported, which brings the reliability of results into question. As well, the test ran for 4 months, which is much longer than the ideal 2-8 week timeframe. Therefore, discretion is advised if you’re planning to use these results to inspire your own testing ideas and practice.


Analysis:

This study provides preliminary evidence that website push notifications may be more effective than email updates at:

  • Promoting your latest content
  • Increasing audience engagement
  • Growing site traffic

Why do push notification seem to work well? There’s likely a couple reasons:

1. Reduce Sign-up Friction

To receive Pushcrew’s notifications, users simply need to click to “allow.” Viewers are then notified when new content is added to the site they’ve subscribed. No email address or personal information is required.

In contrast, subscribing to email notifications is a more arduous process, from users’ perspective. Users first have to consider if they’re willing to divulge personal contact information. They then need to fill-out a form to complete the sign-up. Along the way, there are many opportunities to abandon.

Push notifications seem to remove much of this potential sign-up friction.

2. Make Consuming Content Easier

The notification system also makes it easier to consume content.

Unlike email, readers don’t have to scan through piles of messages, decide if they want to click to open the message, review the content, and then, finally, choose if they want to invest the time engaging on the site.

Readers can simply skim the short notification and click to arrive on the website.

As a result, users may be more receptive to consuming content through this low touch, single click channel.

3. The big HOWEVER

However, it’s important to note, website push notifications are still a relatively new technology. Overtime, users may tire of constant site updates. And, their effectiveness may diminish.

Just as email has become a saturated communication channel, push notifications could suffer this same fate. So, a word of caution: don’t inundate your users with too many notifications.

And, if you’re a content creator, make sure you test the optimal time, frequency, and ways to push your notifications in front of your audiences’ eyes.


Tangible Takeaway & Immediate Application

For now, push notifications appear to provide content marketers – who get in on the trend — with an effective method for building subscriber lists.

However, push notifications are still a relatively new technology. Overtime, users may tire of constant site updates. And, their effectiveness may diminish.


What Do You Think?

Why do you think the version with the push notifications won? Where do you see push notification technology going from here?

Share your thoughts, comments, and questions in the “Comments” section below.

Poll Results - The Best Guesses:

Email or push notifications?

  • Version B
  • Version A
Loading ... Loading ...

Sign-up to get FREE gamified A/B tests and more valuable content delivered to you each week.

I agree to the Terms & Conditions.

Become a More Successful Digital Marketer

Pro Members get exclusive access to the full library of Archived Tests, helpful resources, plus personally connect with industry experts to get your specific questions answered. Get the Pro Advantage

Get Your A/B Test Featured!

Have your A/B test showcased. You'll get a great case study write-up, your logo prominently featured, linkbacks to your site, and social media promo, giving you valuable, free brand exposure. Submit Your Test

3
COMMENTS: Tell Us Your Thoughts.

avatar
2 Comment threads
1 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
2 Comment authors
Deborah OMike W Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Mike W
Member
Mike W

I guessed the push notification (version B) has the highest CTR because with browser push notifications you can be much more relevant.

Other Tests You Might Enjoy

Test Run By

PushCrew

Test Run For

Visual Website Optimizer (VWO)

Test Run On

HubSpot