1. Focus on your goal

Every newsletter should serve a purpose, whether that is to drive revenue (ads or subscriptions), traffic, loyalty/retention, etc. Identify the result(s) that you’re trying to achieve and focus your a/b testing efforts on that.

We launched digital subscriptions to Reuters.com in the fall of 2024. While growing revenue via digital subscriptions is a priority, it’s unrealistic to expect that of every newsletter. Some newsletters are meant to be strong editorial products simply because of their newsworthiness. They’re inbox experiences that serve the readers, but we do expect them to lead to subscriptions and reduce churn. Others do a great job of driving users to the site to eventually hit the paywall and hopefully convert. The formats likely look different, which leaves room to experiment.

Last year, we a/b tested click-centric sections, driven by feeds, to see if we could drive more users to the site from narrative newsletters. This turned out to be extremely effective, increasing click through rate (CTR) across all newsletters. We also tested moving the click-centric section higher up in the newsletter, which always performed better in the a/b test.

The goal of those newsletters is still to generate an engaged and loyal audience, but we were able to find additional value in terms of bringing traffic to the site.

2. Sell your idea and build on the results

Some ideas are more involved and require significant changes or additional resources. You need to justify the change and work. That’s where a/b testing comes in. It’s less of a commitment and a good opportunity to show if the change is worth it.

For example, we were looking to introduce new newsletter sign-up modules and tested one successfully at the bottom of articles. We wanted to move the sign-up module to the top of articles, assuming this would increase newsletter subscriptions, but this would impact ad revenue.

We a/b tested it and found it performed 50% better than the original article sign-up link. Because the test was successful and we realized the impact on ad revenue would be acceptable and the uplift in newsletter subscribers and user engagement would be more valuable, we were able to justify permanently moving up the sign-up link.

Since doing so, clicks on the newsletter sign-up module increased by 63%, which was higher than we expected, with total newsletter sign-ups up by 9%. Our next step is to a/b test our sign-up page to drive more conversions. A/b testing is a continuous effort. You make one small change that influences the next.

> How Reuters and The New York Times use newsletters to drive loyalty, subscriptions, and revenue

3: Keep it simple

Ideas don’t have to be extravagant to matter. The smallest change can move the needle, and a series of simple a/b tests can have lasting impact.

Our earliest tests included basic subject line tests, where we learned that being witty when appropriate while maintaining Reuters standards, tapping into subscriber demographics and utilizing SEO keywords helps increase open rates.

Another test was removing images from list-centric automated newsletters. This increased CTR by 29-33% in four newsletters. We tested moving an image to the bottom of a narrative newsletter rather than the top, which increased CTR by 15%. We also tested the length of a newsletter that summarizes one story per day and found that shorter is better. These tests were nothing fancy, but were still an improvement. A/B testing subject lines, in particular, is useful because the learnings will impact the performance of future editions.

4. Learn from your peers

The best ideas come from our peers. It’s important that the media industry works together by sharing best practices and learnings.

Last year, at The Audiencers Festival, a peer from the Washington Post shared that they saw an increase in clicks after removing descriptions from list-centric newsletters. This was also confirmed by former colleagues at Newsday. We a/b tested this in three of our automated newsletters at Reuters, and increased CTR by an average of 59%. There’s no shame in learning from industry best practices.

5. Listen to your readers

For the relaunch of our flagship newsletter, the Daily Briefing, we had an iterative process that involved prototype-building, surveying the audience, and reflecting regularly whether the changes were successful. Now, two years since we relaunched, it remains one of our most successful newsletters, with consistently high open and click rates. You can review our process in this presentation from ONA 2023.

Our objective was to give the newsletter a clearer design and bring our audience a digestible news overview. We drafted three prototypes of varying lengths and with different sections. Our UX team designed a survey, which more than 300 readers participated in and gave their feedback. The results were clear: a short and concise news overview won over prototypes that had more background information or deep dives into one specific news story. This prototyping experience influenced the development of other Reuters newsletters in our portfolio.

As our strategy continues to evolve, especially with a subscription model in mind, so will our a/b testing, but we have a solid foundation to build from. The point is to learn from the data and to remain curious.

FAZ's Donika Lilova spoke at The Audiencers' Festival London in June 2024 on their digital subscription journey and, in particular, about how they're working on making their paywall dynamic with regular A/B testing to continuously optimize performance. In this article Donika dives deeper into this strategy.

Product + Sales

In 2021, we merged the two departments, Product and Sales, into one customer-focused matrix organisation with mission teams aligned to the customer lifecycle. This restructuring enables us to better understand the needs of our users, cater to their requirements more effectively, make a more efficient use of our internal resources and reach our set goals in a structured manner.

The Paywall Team

My team is responsible for generating new subscribers through the paywall on the website and in our apps. Our team is cross-functional, comprising colleagues from the editorial, data, online marketing and sales, and product management teams.

We believe that digital growth is generated from the product, and our paywall team plays a vital role in directly contributing to our company strategy on the website and in the apps.

FAZ+

Our paid content offering, FAZ+ represents the most significant contribution to our digital growth. It’s available to all users on FAZ.NET, providing access to over 1,000 paid articles with a straightforward entry and exit process.

We have successfully driven consistent and sustained growth for this offering, reaching nearly 140,000 paying digital subscribers in six years. Our digital growth remains on a strong upward trajectory.

Our core team DNA is formed by our test and data-driven product development and digital marketing activities. A/B testing is a central component of our day-to-day operations. It took us several years to develop a streamlined testing process with standardized forms and to build up a knowledge database with all the conducted tests and learnings. Our Product + Sales department currently manages around 60-70 A/B tests per year and our goal is to be able to fit in even more parallel tests within our testing pipeline.

Our objective on the paywall is to increase clicks and conversions in order to optimize our standard and increase customer lifetime value. In 2018, we began with two to five paid articles using our freemium paywall model. Since then, we have increased the number of premium articles and tested more dynamic paid content approaches for different audiences and segments, based on their propensity and willingness to subscribe.

> An essential read: How to dynamic paywall (it’s a lot more simple than you think)

Dynamic Paywall Test

One of our more prominent tests worth mentioning is our Dynamic Paywall Test, which was conducted in 2023 and examined different engagement segments.

Test Hypothesis

The following test hypothesis forms the basis of this experiment:

If we increase the number of Paid Articles on our desktop and mobile websites, 

then we will generate more paid subscriptions, 

because the users see more paid articles and thus, a higher buying pressure arises.

Test Setup

The test was conducted as an A/B test with an even audience split. Our objective was to focus on desktop users. This year, we conducted another re-test for mobile users only and were able to confirm the overall tendency. The test was deployed using our CMS and Adobe Target, with subsequent data analysis conducted using Adobe Analytics and Microsoft Power BI.

A total of 30% of the content used in the test was free, meaning that users would consistently view these articles as such.

Forty percent of the content was paid and displayed as premium content with the F+ icon and a paywall.

The remaining 30% were dynamic articles, which were displayed as either free or paid content, depending on the user segment.

Learnings

The key takeaways from our main test are as follows:

1. The introduction of a greater number of premium paid content items has been found to result in an uplift of 48% for users who demonstrate a higher level of engagement. This equates to approximately 20 additional paywall orders per day.
2. The A/B test presented here demonstrates the importance of addressing our users on a granular level.
3. It’s recommended that users with a low level of engagement continue to be offered the option of a lower paid content subscription, with the aim of increasing their engagement levels and subsequently transitioning them to a paid subscription model.

Test Results (Detail)

	Unique Visitors	Conversion Rate Orders Clicks		Conversion Rate Orders Overall
Control	50%	1,06%		0,11%
Variation	50%	1,46% significant winner	Uplift: 38,05% confidence: 100,00%	0,16% significant winner	Uplift: 48,08% confidence: 100,00%

We rolled out this test successfully at the end of 2023/ beginning of 2024 with a subsequent follow-up test conducted exclusively for mobile users. The results of the subsequent test also corroborated the initial hypothesis. There was a notable increase of nearly 10% in orders when more paid content was distributed via mobile.

What are our next steps?

In June 2024, we successfully conducted our first A/B Test Dynamic Paywall Offer based on a propensity-to-buy score we internally developed with our data team. The results there were very promising, which is why we are continuing our paywall journey towards scoring models and dynamic offers, while also optimising our paid content.

A/B testing is essential to being successful in the subscription business. Whether Spotify, Netflix, Disney or The New York Times - everything is tested before it is rolled out on a large scale.

I have conducted over 100 A/B tests in the last few years and would like to share my top 4 learnings. Although these insights come from a non-data perspective, they are particularly valuable for anyone approaching the topic from a product or growth perspective. My learnings should serve as an impulse for you to delve deeper if you are interested.

1. Pay attention to the novelty effect

A common stumbling block in my A/B testing was the exciting first uplift. After a week of testing, you look at the initial results and see a big uplift in the test group.

But be aware that this gain often disappears after a few weeks, a classic sign of the novelty effect. Returning users in particular can be tempted to behave atypically in the short term in reaction to new elements such as a new paywall layout or different features.

To ensure you don’t fall for such test results, I recommend:

• Extending the testing time: A longer testing period helps distinguish initial excitement from real improvements. A declining difference between the test and control groups over time indicates the novelty effect.
• Analyze first-time visitors: If first-time visitors don’t show any significant differences between the test and control groups, but returning visitors do, then take results with a pinch of salt.

> You’ll also be interested in: A/B testing paywall benchmarks

2. Determine the test size in advance

It’s frustrating to discover that test and control groups were not large enough to provide valid results – especially when the test has already been completed. To prevent this, it’s crucial to calculate the ideal test size for your A/B test in advance.

The need-to-know about significance:

Not every change is an improvement. Which is why it’s important to understand statistical significance:

• Significance level: By default we use 5%, which means that we have a 5% probability of assuming an effect that doesn’t actually exist (Type I error). For smaller volumes, a level of 10% or lower may also be acceptable.

Tools like abtestguide.com allow you to check the significance of your results and experiment with different levels.

To determine the size of your test group before starting the test, Evan Miller’s sample size calculator will help you. This saves you a rude awakening later in the test, and also helps you to plan the timing of your A/B tests.

As an example, let’s assume you want to test which subscription benefits to promote on your paywall. This should increase your standard 1.5% conversion rate by at least 10%. When determining your test size, you take the following into account in Evan Miller’s calculator:

• Base rate: The standard form of the KPI to be considered. In this case 1.5% (conversion rate).
• Minimum Expected Improvement: The smallest effect you still want to see, relative to the base rate. Here, 10%.
• Alpha: The chosen significance level, 5% or 10% depending on your risk tolerance.
• Beta: The test power, usually at 80% (Beta = 0.20), indicates the probability of not detecting an effect even though one actually exists (Type II error).

For our example you would have to fill out the calculator like this:

It’s even easier with ChatGPT! Prompt for this:

My conversion rate is 1.5%. I hope my measure will result in an improvement of 10%. Tell me, according to Evan Miller, how large my test group needs to be at 5% significance.

3. One KPI per test

In every A/B test you should focus on just one KPI. Decide on a specific metric for each test, be it increasing conversion rates, improving click-through rate on the paywall, increasing dwell time, or page views per visit.

I know from personal experience that it’s tempting to look for positive signals, especially when a test doesn’t show clear results. We all desperately want our efforts to be rewarded. But be careful – the more KPIs you evaluate in an A/B test, the higher the probability of finding random positive deviations.

Let’s take our A/B testing example from above. You’re running a test to see if a new paywall layout, with different subscription benefits, will increase conversion rates. In this case, conversion rate should be your only KPI. If you also look at other metrics such as click rate or dwell time, you increase the risk of mistakenly identifying positive effects.

This risk is known as the Family Wise Error Rate (FWER). If you check the significance at 5% in an A/B test and only focus on one KPI, the risk of making a Type I error is a maximum of 5%. However, if you test seven different KPIs in the same test, the risk increases of around 30.17% that at least one of these KPIs will be incorrectly considered significant:

FWER = 1 - (1 - 0.05) ^ 7 ≈ 30.17%

This highlights the importance of focusing on just one KPI in each A/B test to ensure methodological cleanliness and statistical accuracy.

4. Alternative option

Not every publisher has the opportunity to mobilize enough users for significant A/B tests. And honestly, no one wants to run a single test for 3 months or longer. Larger publishers also face this challenge for tests further down the funnel, such as those around churn.

But before you rely completely on your gut feeling, usability tests can be a helpful alternative. They don’t provide a clear statement as to whether a measure will bring X percent more conversions or less churn, like A/B tests do. However, they do offer valuable insights to make more informed decisions.

During a usability test, you observe and analyze how users complete certain tasks. You receive feedback directly through surveys or indirectly by observing their reactions.

Let’s take our familiar example again. You guide users through different paywall versions (with and without subscriber benefits listed on the wall) and collect feedback on clarity, persuasiveness and user decisions. Typical tasks could be:

1. Evaluate paywall information
   • “Consider the information on the paywall. Please let us know what you think about the information presented and whether or not it influences your decision to proceed.”
   • Goal: Gather feedback on the clarity and persuasiveness of the information on the paywall, both with and without benefits.
2. Decision-making process
   • “Please decide whether you would click the button on the paywall to gain access. Explain your decision to us.”
   • Goal: Understand user decision-making and the factors that influence it, especially the influence of benefits.
3. Compare the variants
   • “Here are two versions of our paywall: one with benefits and one without. Please compare them and let us know which version makes you more likely to click and why.”
   • Goal: Directly compare the two versions to determine which is more effective in terms of user experience and increasing click-through rate.

5-8 conversations are often enough for meaningful results. Tools like maze and rapid user testing can be helpful for unmoderated tests.

By the way, usability tests and interviews also ideally complement A/B tests. They not only help you with optimization, but also provide information about why users behave the way they do.

I hope that these learnings are useful for your future testing projects and wish you much success in your next tests.

Speak German? Subscribe to Sascha's blog, Sub Growth or follow him on LinkedIn!  "In this blog, once a month I take a fresh and in-depth look at proven growth tactics that are specifically designed to advance the subscription business of German newspaper publishers. By sharing tried-and-tested frameworks, interesting benchmarks and exciting best cases, I want to provide practical and applicable insights that will help you in your work."

"Making decisions is easy, what’s hard is making the right decisions."

Netflix, the highest-earning media service provider on the market. A hard subscription model that’s converted over 220 million paid subscribers. At least 17,000 titles across all its international libraries… It’s clear that Netflix is doing something right! 

For one, Netflix is constantly evolving product offerings and quality of experience to improve the value offered to subscribers.  

Just look at Netflix in 2010 compared to today:  

In 2010, static user interface, limited navigation options and a layout inspired by video rental stores

Fast forward to 2022 and Netflix offers an immersive user interface with a video-forward experience, richer navigation options and beautiful box art that personalizes to the user. 

But this transition didn’t happen overnight. It’s required countless decisions. And whilst making decisions is easy, what’s hard is making the right decisions.

How can Netflix be sure that their decisions will lead to a better product experience for existing members and support the acquisition of new ones? 

The answer – A/B testing. 

As the Netflix Technology blog put it, whilst decisions made in a team are limited to the viewpoints and perspectives of those involved, A/B testing is a form of experimentation that gives subscribers the opportunity to vote subconsciously with their actions on how to evolve the Netflix experience. 

More specifically, A/B testing allows the teams to prove causality. Putting knowledge into more confidently making changes to the product knowing that subscribers have voted for them with their actions. 

It also plays a key role in implementing an iterative learning process, alternating between deduction and induction to continuously optimize performance. 

George Box’s diagram shows this process from deduction, going from an idea or theory to a hypothesis to actual observations/data that can be used to test the hypothesis, to induction, the process of generalizing from specific observations/data to new hypotheses or ideas.

Running these experiments

The hypothesis: If we make change X, it will improve the member experience in a way that makes metric Y improve.

Take a random sample subset of subscribers (the reasoning behind ‘random’ is explained below), evenly splitting this sample into two groups – the control and the treatment. 

Both will receive a different experience based on the defined hypothesis. 

Wait for the test to run its course before comparing the two groups based on a variety of predefined metrics.  

Think carefully about your chosen metrics and what they convey

High click-through rate, for instance, may well imply that this flipped image is more inviting and entices more users to watch the title. However, it could also mean that the text is harder to read, so subscribers click on the title to read it more easily. 

In this case, it might be valuable to also measure the percentage of subscribers who then leave that content after clicking vs those who go ahead and watch it. 

Hold everything else constant

Coming back to why Netflix randomly selects users for their two groups (control and treatment), this plays an essential role in ensuring the groups are balanced on all dimensions that may influence the test results. 

For instance, it ensures that the average length of Netflix subscription isn’t significantly different between the groups, nor that they have varying content preferences, primary language selection, etc. The only difference is the product experience being tested, meaning results clearly prove or disprove the hypothesis. 

“A/B tests let us make causal statements. We’ve introduced the Upside Down product experience to Group B only, and because we’ve randomly assigned members to groups A and B, everything else is held constant between the two groups. We can therefore conclude with high probability…that the Upside Down product caused the reduction in engagement.”

Netflix Technology Blog

A real-life example from Netflix – it all starts with an idea

Did you know, the ‘Top 10’ titles section of your Netflix home screen was once just an idea that turned into a testable hypothesis. 

The idea: surfacing titles that are popular in each country would positively impact our subscribers experience in 2 ways

• By presenting the most popular titles, Netflix could help members have a shared experience and connect with each other through conversations
• Supports subscriber’s decision-making process for choosing what to watch by fulfilling the intrinsic human desire to fit-in and be a part of a shared conversation

Turning the idea into a testable hypothesis: 

Showing members the Top 10 experience will help them find something to watch, increasing member joy and satisfaction.

Choice of a primary decision metric: member engagement with Netflix (made up of a few metrics)

Secondary metrics included title-level viewing of those titles that appeared in the top 10 list, the percentage of viewing that originated from that ‘Top 10’ row vs other parts of the page, etc. 

What’s more, the team always adds on some Guardrail metrics to limit any negative consequences that could arise from the test, as well as spot any product bugs. For instance, they might compare the number of contacts to customer service for the control and treatment groups, checking that the new feature isn’t increasing contact rate, which could indicate subscriber confusion or dissatisfaction.

Optimizing your paywall and subscription strategy to compete for a place in your reader’s monthly budget is no mean feat. And whilst there are endless benchmarks and success cases online, there is really only one way of finding the best performing strategy for your unique audience: testing.

Being one of the most user-friendly but efficient ways of optimizing your paywall performance, testing is something that is highly recommended for publishers to do on a regular basis and in a variety of forms.

But, unlike many of the other optimization methods, it is not the easiest to benchmark given that the whole point of running an A/B test is that users are unaware when one is taking place.

Here are three examples of paywall A/B tests run by digital publishers – what can you test, what do you need to define prior to launching the test, what were the results and, most importantly, what can we learn from these to apply to your own tests?

Bio à la Une

This French health and well-being publisher, who only recently launched a premium strategy, wanted to learn more about their audience and what makes them convert. They, therefore, ran a series of tests, one of these varying the wording on the call-to-action button.

Hypothesis: Click-through rates (CTRs) will increase by using a more direct call-to-action phrase, “je m’abonne!” (Subscribe), one tha is commonly employed by digital publishers in France.

Variable: Text on the CTA (call to action) button.

Goal: Increase CTRs and discover the focus that works best for each audience segment.

Control (version A): CTA button text “Discover our offers”, a more invitational, indirect phrase.

Treatment (version B): CTA button text “Subscribe” takes a very direct approach.

Results: Version B had a higher click-through rate amongst less engaged users, which suggests that these audiences respond better to a direct CTA phrase. Version A, which invites readers to discover subscription offers, performed better for more engaged users.

This was an interesting result, and one that highlights the importance of segmenting your audience before testing. In this instance, the publisher used Poool’s Dashboard to segment users based on level of engagement (Volatiles, Occasionals, Regulars and Fans) which gave insights into how these users respond differently to direct vs indirect wording on the paywall.

To further optimize your paywall performance and gain a deeper understanding of user behavior, you could also consider segmenting audiences based on location, device or even a users interests if you’re able to gather this information.

The takeaway: Just as the performance of a paywall will differ on your site compared to that of another publisher, this can also be said for one segment of your audience compared to another.

ELLE Magazine

The fashion and beauty magazine launched its digital premium strategy with a two-step conversion journey consisting of a registration wall followed by a hard paywall. This proved hugely valuable for increasing engagement and informing their audience of the concept of paying for access to content, but the team wanted to test a more direct approach to see if this led to higher conversion rates.

Hypothesis: A hard paywall will lead to higher conversion rates compared to a soft engagement journey.

Variable: User journey.

Goal: Increase conversion rates.

Control (version A): A two-step scenario with a soft conversion registration wall followed by a paywall.

Treatment (version B): A hard paywall alone.

Results: Version B led to slightly higher conversion rates than version A. However, these results have to be taken with a pinch of salt…

Firstly, it is important to consider the stage of your subscription model. For ELLE, although version A worked extremely well for launching its strategy, collecting data and increasing engagement prior to subscription, version B performed better when this test took place, meaning their audience was perhaps more qualified and less likely to be frustrated by a hard paywall.

You also have to consider more than simply conversion rates as a KPI (key performance indicator) when altering your user’s journey, namely ARPU (average revenue per user). Considering engagement has a direct correlation with revenue, and that soft conversion steps such as registration walls increase user engagement, it is arguably more beneficial in the long term to employ a user journey such as version A, despite what these test results suggest.

Specifically, de-anonymizing a user through registration increases ad-based revenue thanks to targeting and also boosts propensity to subscribe (and increases the likelihood that this user will remain subscribed in the long run).

The takeaway: When looking at benchmarks, do not forget to consider the context surrounding them, take overall ARPU into account and remember that it is not only the paywall that influences conversion rates.

For instance, although ELLE did choose to employ journey B on all user segments, it also worked on increasing the visibility of premium content, reducing frustration by adding yellow tags on these subscriber-only articles as well as boosting engagement in other areas, such as by promoting their newsletter and membership offers. These efforts had a big impact on balancing frustration and engagement which ultimately support high conversion rates.

Neos Kosmos

One of the most important aspects of your premium model is the value proposition. Why should a user subscribe? What pain points will it solve? How will it help the reader achieve their ‘jobs-to-be-done’?

And, of course, this value proposition needs to be clearly communicated to readers on your paywall in a way that is easily understood and gives enough encouragement to click-through and subscribe.

This is something that the Australian-Greek publisher Neos Kosmos understands well, leading them to test out a shorter version of their value proposition which focuses more on ‘supporting’ the publisher than on the benefits of subscribing.

Hypothesis: A value proposition focusing on ‘supporting’ Neos Kosmos will lead to higher conversion rates.

Variable: Wording on the wall (value proposition).

Goal: Increase conversion rates.

Control (version A): The original wording details what Neos Kosmos’ premium offer gives a subscriber access to.

Treatment (version B): This version takes a more ‘support our work’ approach, appealing to a user’s emotional and cultural attachment to Greek culture in Australia.

Result: Version B led to a 16 per cent increase in conversion rates.

For Neos Kosmos, this showed the importance of appealing to their audiences’ cultural heritage and the shared values amongst readers. For many publishers, taking the angle of asking for support rather than providing access to content is hugely valuable for increasing conversion rates and developing a community of loyal subscribers.

The takeaway: Value propositions need to be clearly defined prior to launching your premium strategy but should also be tested on a frequent basis to discover the optimal angle, wording, etc. for your unique audience.

To summarize:

• Clearly define your hypothesis, variable, goal and audience segments prior to launching the test
• Remember that there is more at play than just the paywall, such as value proposition, visibility of premium content, the balance between frustration and engagement, etc.
• Tracking click-through and conversion rates alone has its limitations – ARPU is arguably the most important
• As always, benchmarks are great and useful for inspiration, but there is no one-size-fits-all strategy that will work for every digital publisher

Happy testing!