guess_the_test-white-green
Member Login
guess_the_test-white-green
Menu

The Complete Guide to A/B Testing in a Cookieless World

By: Deborah O’Malley and Joseph A. Hayon | Last updated January, 2022

Overview

Did you know that 96% of adult users report being concerned about their privacy on the Internet? And 74% say they’re wary of how their privacy and personal data is being used.

With rising fears over Internet privacy and security, there’s been a push to protect users and tighten privacy policies.

As a result, 3rd party cookies will soon be blocked by most major web browsers, leaving optimizers and experimenters desperately picking up the crumbled pieces.

We'll have no easy way to track user events or behaviors -- let alone measure conversions.

What's an experimenter to do?

In this article, you’ll learn everything you need to know about cookies, their implication on A/B testing, and how to ensure continued success in your testing practice.

Specifically, you’ll find out:

  • What cookies are
  • The two most important types of cookies
  • How a cookieless world may impact on A/B testing
  • Server-side tag management and what it means for your testing practice
  • What you should do to adapt to the changing landscape

What are cookies? 

Beside those delicious, tasty snacks, what are cookies anyway?

Well, in the context of the World Wide Web, cookies are text files sent from a website to a visitor’s computer or other Internet connected device.

That text file identifies when a visitor is browsing a website and collects data about the user and their behavior.

Collected data may include elements like a user's personal interests, geographical location, or device type.

These data points are stored on the user's browser. The information storage occurs at the time the user is engaging with a website, app, or online ad.

Through this collection and storage process, cookies achieve three things.

They can:

  1. Gather and store information about the user
  2. Capture the actions or events a user takes on a website or app
  3. Track a user’s activity across multiple websites

Cookies – the good, the bad, and the ugly

The good

Cookies attempt to provide users with a more personalized experience when engaging with a website or app.

When marketers analyze tracking data, it can provide them with guidance into where optimizers' focus should be placed. The insights available are often richer than what standalone spreadsheet results or visual dashboards can provide.

So, despite their bad rep, cookies can, in fact, help digital marketers detail what, where, and why certain web events happen.

For these reasons, cookies, can be a valuable (and delicious) part of the web.

The bad

However, because cookies track and store a user’s browser activity, they’ve been demonized as invasive and intrusive monsters. Cookie monsters, that is.

The ugly

With growing concerns over user privacy, there’s been a call to arms for the abolishment of cookies.

Getting rid of cookies may spell bad news for digital advertisers, marketers, experimenters, optimizers, and A/B testers reliant on using tracked data to understand user behavior and measure conversions.


The two types of cookies you need to know

While there are plenty of tasty cookies out there, as a digitals marketer, there are really only two kinds you need to know: 1st and 3rd party cookies.

The difference between these two types of cookies is based on how a user's browsing activity is collected and where that data gets sent. 

1st Party Cookies Defined

1st party cookies are directly stored by the website or domain you visit or the app the user has logged into.

Once the user’s data has been captured, it is sent to an internal server.

3rd Party Cookies Defined

3rd party cookies have a slightly different flavor.

3rd party cookies are created by outside domains and relayed back to an outside third-party server, like Google, Facebook, or LinkedIn.

It's because the data is collected and sent out, to a 3rd party, that 3rdparty cookies are so named.

While this information might be interesting, as a digital marketer, what you really need to know is, 3rd party cookies are the troublesome ones of the batch.

They’re essentially tracking pixels.

And because they track and send data back to advertisers’ servers, they're seen as intrusive and highly invasive of user privacy.

For this reason, privacy protection advocates don’t like them and want them abolished.

And so, big-name brands, like Google and Facebook, are being forced to shift into a cookieless world -- one where 3rd party cookies will no longer exist.

In fact, Google is planning to eliminate the use of 3rd party cookies as soon as early 2022 and will likely move instead into privacy sand box initiatives.

How do 1st and 3rd party cookies work?

Now that you intricately understand what a cookies is, and the differences between a 1st and 3rd party cookies, you can explore the intricacies between how a 1st and 3rd party cookie works.

Grasping this concept will help you aptly shift from a reliance on 3rd party cookies to adopting other solutions that still enable the data tracking, collection, and conversion measurements you need for A/B testing.

The inner workings of a 1st party cookie

To fully understand how a 1st party cookie works, you have to start with realizing the internet has what’s known as a “stateless protocol.”

This term means, websites inherently “forget” what a user has done, or what activity they’ve taken -- unless they’re given a way to remember this information.

1st party cookies solve this forgetfulness problem.

They track and capture user activity information and, in doing so, aim to deliver a highly identifiable and authenticated user experience.

For example. . .

Imagine a user goes to your eCommerce website, adds an item to their cart, but doesn’t end up making a purchase during their initial session.

Well, the cookie is what enables the items added to cart to be saved in the user’s basket.

So, if the visitor later returns to your site, they can continue their shopping experience exactly where they left off --with the items still in their basket from the previous session.

All thanks to 1st party cookies.

This process can aid users and provide a better user experience.

So, you can see, 1st party cookies really aren’t that bad!

And, the good news is, most everyone seems to agree, 1st party cookies are a-okay. So they're here to stay. At least for now. . .

How 3rd party cookies work

An easy way to understand  how 3rd party cookies work is by imaging this scenario:

Pretend you’re searching online for a local sushi restaurant and land on “sushi near me” website.

But then, your kid abruptly grabs your phone out of your hand and insists they play you their latest, favorite Facebook video.

It’s 3rd party cookies that will have tracked this behavior and try to serve you ad content believed relevant based on your browsing activity.

The mystery for why you're getting Nyan Cat memes in your Facebook feed is now explained. . .

And, 3rd party cookies are also the reason why you might get served Facebook ads for that local sushi restaurant you just searched.

What happens to user data after a 3rd party vendor gets ahold of it?

After the data is relayed back to the 3rd party, it’s then pooled together by all the data warehouses owned by the various ad tech vendors.

This data collection and pooling process is what enables marketers to:

  1. Deliver targeted ads relevant to users’ interests
  2. Develop look-a-like audiences
  3. Build machine learning processes aimed at predictive ads
  4. Control the number of times a user sees an ad through any given ad network

With these capabilities, it’s possible to measure ad effectiveness, assess key conversion metrics, and gauge user engagement across multiple website networks.

None of these assessments would be easily possible without 3rd party cookies. Which means, without them, A/B testing may become significantly difficult!


How a cookieless world may impact A/B testing

If you’re concerned about protecting your data and privacy, the abolishment of 3rd party cookies may be welcomed news.

But, if you’re like most marketers, you might be a little freaked out and wondering how the abolishment of 3rd-party tracking cookies is going to impact the tracking and data collection capabilities in your Conversion Rate Optimization (CRO) and A/B testing work.

How will the abolishment of 3rd party cookies affect optimizers?

The answer is -- likely in three main ways:

  1. Legislatively
  2. Through cookie banner requirements
  3. Within browser tracking

Let’s look at each concept in more depth. . .

1) Legislative Reforms

When the Internet surfaced to the mainstream in the late 1990’s, www, might have just as well stood for wild, wild west.

There weren’t many laws or privacy controls enacted back then.

But, as the web has matured, more privacy policies have become instituted.

GDRP requirements were just the starting point. As Stape.io states,

“Governments are creating laws and regulations designed to encourage tracking transparency. These laws are geared towards preventing companies from unethical or unregulated user tracking. In a nutshell, it's all for the sake of user privacy. As a result of GDPR, CCPA, and ePR, websites that don't reveal which user data is collected or how it's used can face civil penalties.”

Here's a timeline showing recently implemented privacy policies:

From a legal perspective, digital marketers are being required to comply with more, and more aggressive, data-privacy regulations.

And this trend will likely only continue as time marches on.

2) Cookie banner requirements

Website owners must now also provide transparency at the start of a user’s journey by prominently displaying a cookie banner on their website.

We’ve all seen these cookie notification banners. They look something like this:

Now you know why they’re on so many websites! 🙂

And, while cookies banners may slightly disrupt an A/B test experience, they're not the worst offender.

3) Browser tracking

Worse yet are increasingly stringent browser privacy policies.

For example, Apple’s Safari browser recently instituted aggressive policies against cookie tracking.

Safari now “kills” cookies after 7 days, and kills cookies from ad networks after just 24 hours!

These timelines are a big problem, especially for testers following best practice and running an experiment for +2 weeks.

Why?

Well, let's say, for example, a Safari user enters a website being tested, leaves the site and comes back 8 days later?

By this time, the cookie has been dropped, and, as a result, the user is now classified as a new visitor and may see a different test variation.

The result: the experiment data may be dramatically skewed and unreliable.

What's going to happen to data collection with A/B testing?!

If this kind of scenario sounds doomsday to you, you might be wondering how you're now possibly going to accurately track and measure browser behavior, events, or conversions.

Here’s the likely outcome:

Fragmented data collection

Data collection and measurement will likely become more fragmented.

As 3rd party cookies are phased out, experimenters will be forced into relying largely on 1st party cookie data.

But, this data will be accessible only from users who consent to the site’s cookie policies and data collection or login terms.

Since fewer users are likely to accept and consent to all privacy policy, data profiles will become more anonymous and less complete.

Net sum: there will be big gaps in the data.

These gaps will likely only grow larger as privacy laws evolve and become more stringent.

Modeled user behavior

Currently, the digital marketing realm seems to be moving towards more of an event-style tracking model, similar to Google Analytics 4 (GA4) data collection reports.

This movement means, in the end, we’ll have more algorithms pointing to modeled user behavior -- rather than actual individual user data.

And, we’ll likely need to rely more on data modeled through statistical calculations, methodologies, and technologies.

Essentially, we’ll move from an individual data collection approach to an aggregated, non-user-segmented overview.

This approach will work if we compile enough data points and invest in developing robust predictive analytics data collection and measurement infrastructure.

Untrackable user data

As privacy policies rachet down, tracking user behavior is going to get far more challenging.

For example, Apple recently released its newest operating system, iOS 15.

Built into this OS are a host of security measures that are great for user privacy, but prevent the tracking of any user behavior.

In fact, right now, the only new Apple device without privacy controls are the Airpods!

In addition to Apple, other browser brands are also cracking down. As such, an anonymous user browsing experience -- with untrackable data -- is becoming the norm.

In turn, private, secure browsers, like Brave, will likely become mainstay.

Meaning, yet again, less data and metrics available to marketers.

The final outlook?

But, in the end, A/B testing likely won’t die. It will just change.

Conversions are most likely going to be based on broader machine-driven learning algorithms that model users, predicts and analyzes expected behavior, then provides anticipated outcomes.

Data will be less about observed insights and more about predicting how we think users are likely to behave.

Which, when you really get down to it is kinda like how most experimenters declare A/B test winners now anyway. 😉


Server-side tag management

Big changes are coming.

Major privacy and security enhancements will shift the way experimenters track, access, and measure conversion data.

3rd party cookies will soon be abolished, and marketers will be left with big holes in their data collection and analysis capabilities.

For data purists, the situation sounds a bit scary.

But, the good news is, server-side tag management may be a potential solution. 

What is server-side tag management is, how does it work, and how might it help A/B testers?

What is server-side tag management?

According to data wizard, Simo Ahava, server-side tag management occurs when you run a Google Tag Manager (GTM) container in a server-side environment.

If that terminology doesn't mean much to you, don't fret.

Simply stated, it means server-side tagging is a bypass around 3rd party cookies.

How does server-side tag management work?

You'll recall, with a 1st party cookie, the cookie is deployed by the owner of the website domain.

Well, with server-side tracking, the data is sent to your own server infrastructure. A cloud server is then used to send data back to 3rd party platforms.

Doing so gets around tracking restrictions and privacy laws. At least for now.

How does server-side tag management help A/B testers?

This solution helps enable experimenters to take more control of their data. With server-side tag management, you can choose how to prep and clean the data as well as where to unload users' data.

As a result, there are 3 main advantages. With server-side tagging, you can continue to:

  1. Serve 3rd party content
  2. Achieve better, faster site performance
  3. Reduce data gaps

Let's explore each advantage in a bit more depth. . .

1. Serve 3rd party content

In server-side tag management, typically, the data is ported into a 3rd party ad network. This data flow means, marketers can serve and target content based on users’ indicated interests. 

Marketers can also mask the identity of the browser by changing the container name or hostname.

As a result, an adblocker won't recognize it as something trying to identify the users, so the ad won't be as redily blocked. 

2. Achieve better performance

From a performance standpoint, server-side tagging is faster and more efficient because less code needs to load on the client-side.

So, A/B tests may be able to run more quickly potentially without lag or flicker.

3. Reduce data gaps

Server-side tag management can also prevent the data from "leaking" or getting lost in the shuffle.

But, most importantly, server-side tag management can reduce the gap between conversion events and actual user behavior.

Final thoughts on server-side tagging

While the shift to server-side tagging may not be the end-all solution, it is a means currently available to help digital marketers take better control of how user data is captured.

For now, it's a viable solution that's worth investing and learning more about.


Adapting to change

To stay ahead of the curve, here are the top 3 things you can do in this changing environment as we shift into a cookieless world.

1. Build your relationship with your customers

Don’t just rely solely on your business web analytics data.

Learn about your users.

Really get to know your audience. Find out what they like and what keeps them away from your site.

You don’t need to rely on analytics data to do so.

You can talk to your customers, poll them, apply qualitative data or do User eXperience (UX) studies to learn about your customers their needs, wants, desires, and how they behave on your website.

2. Get to know your analytics account

Audit the data collection integrity of your analytics data one or twice a month depending on your organization’s needs.

Note any visible changes, and try to dig into elements that are less guided by analytics tools. 

3. Stay up to date

Keep up with new updates to ad tech platforms, like Facebook’s new API implementation for tracking Facebook events, as well as updates with GTM server side tagging.

Some good resources for staying on top of the latest trends include Simo Ahava’s blog and Google’s Developer Resources.

Summary

We're moving into a cookieless world where user privacy and personal data protection are paramount.

This paradigm shift may present many challenges for data-driven marketers reliant on tracking users to personalize the customer experience and accurately calculate conversions.

Despite the changing landscape, as an experimenter, you can still continue to build relationships with your customers within a first-party environment.

Also remember, as data collection methods change, you can’t expect to achieve 100% data accuracy.

Rather, you’ll be using more sampled data to piece together an approximation of how customers behave.

In the end, A/B testing likely won’t die. It will just change.

For even more information about A/B testing in a cookieless world, checkout this webinar replay.


Hope you’ve found these suggestions useful and informative. Please share your thoughts and comments in the section below. 

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments

Other Posts You Might Enjoy

eyechevron-downtwitterfacebookarrow-circle-upenvelopelinkedinxingpaper-planepinterest-pwhatsappcommentingmagnifiercrossmenu-circlecross-circle
linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram