Background & Test Details:

Pushcrew is a push notification platform that lets you send interactive, clickable updates to your subscribers, leading them directly to your website.

Pushing the boundaries, Pushcrew’s testing team ran this notable test for the blog section of their sister site Visual Website Optimizer (VWO), the A/B testing platform. VWO’s blog covers various Conversion Rate Optimization (CRO) and testing topics.

The test goal was to determine if more readers would engage with new blog content when sent updates through email or push notifications.

Hypothesis:

The team suspected sending push notifications would increase reader engagement, since the notification system would facilitate a quicker, easier sign-up process.

Specifically, the testing team expected:

• Clickthrough rates of promoted blog posts would increase through website push notifications, compared to email notifications.
• Subscriber list rates would grow faster using push notifications, than through email sign-up.

Test Set-up:

To determine the effectiveness of browser-based push notifications, the PushCrew team assessed sign-up rates across 13 different blog posts, published on the VWO blog site.

Readers were either sent an email, push notification, or both. The email template was formatted to look like this:

The same 13 blog posts were then sent, to push notification subscribers. The push notifications looked like this:

There was a strong overlap between the email and push notification distribution list. If users opted into both email and push notifications, they received both messages. However, push notifications were sent a week later than the email, to reduce reader fatigue.

This content distribution process continued for 4 months. During this time, over 35,000 subscribers received either emails and/or push notifications.

Variations in clickthrough and subscriber list rates were tracked between the two communication channels, using HubSpot.

The Real-Life Results:

Winner: Version B – as the team expected, push notifications pushed conversions much higher.

With push notifications, clickthrough rates increased an astounding 331%, compared to email!

And, the push notification list grew 8 times more than the email list.

However, once visitors clicked to the content, there was no difference in the amount of time spent on the page. This results shows the notification system only impacted the likelihood of whether readers would click to the content – not how engaged they were once reading.

How Trustworthy Are The Results?

This outcome seems moderately trustworthy, based on the large sample size of 35,000 subscribers. However, it should be noted, no confidence level was reported, which brings the reliability of results into question. As well, the test ran for 4 months, which is much longer than the ideal 2-8 week timeframe. Therefore, discretion is advised if you’re planning to use these results to inspire your own testing ideas and practice.

Analysis:

This study provides preliminary evidence that website push notifications may be more effective than email updates at:

• Promoting your latest content
• Increasing audience engagement
• Growing site traffic

Why do push notification seem to work well? There’s likely a couple reasons:

1. Reduce Sign-up Friction

To receive Pushcrew’s notifications, users simply need to click to “allow.” Viewers are then notified when new content is added to the site they’ve subscribed. No email address or personal information is required.

In contrast, subscribing to email notifications is a more arduous process, from users’ perspective. Users first have to consider if they’re willing to divulge personal contact information. They then need to fill-out a form to complete the sign-up. Along the way, there are many opportunities to abandon.

Push notifications seem to remove much of this potential sign-up friction.

2. Make Consuming Content Easier

The notification system also makes it easier to consume content.

Unlike email, readers don’t have to scan through piles of messages, decide if they want to click to open the message, review the content, and then, finally, choose if they want to invest the time engaging on the site.

Readers can simply skim the short notification and click to arrive on the website.

As a result, users may be more receptive to consuming content through this low touch, single click channel.

3. The big HOWEVER

However, it’s important to note, website push notifications are still a relatively new technology. Overtime, users may tire of constant site updates. And, their effectiveness may diminish.

Just as email has become a saturated communication channel, push notifications could suffer this same fate. So, a word of caution: don’t inundate your users with too many notifications.

And, if you’re a content creator, make sure you test the optimal time, frequency, and ways to push your notifications in front of your audiences’ eyes.

Tangible Takeaway & Immediate Application

For now, push notifications appear to provide content marketers – who get in on the trend — with an effective method for building subscriber lists.

However, push notifications are still a relatively new technology. Overtime, users may tire of constant site updates. And, their effectiveness may diminish.

What Do You Think?

Why do you think the version with the push notifications won? Where do you see push notification technology going from here?

Share your thoughts, comments, and questions in the “Comments” section below.