A/B testing is essential to being successful in the subscription business. Whether Spotify, Netflix, Disney or The New York Times - everything is tested before it is rolled out on a large scale.

I have conducted over 100 A/B tests in the last few years and would like to share my top 4 learnings. Although these insights come from a non-data perspective, they are particularly valuable for anyone approaching the topic from a product or growth perspective. My learnings should serve as an impulse for you to delve deeper if you are interested.

1. Pay attention to the novelty effect

A common stumbling block in my A/B testing was the exciting first uplift. After a week of testing, you look at the initial results and see a big uplift in the test group.

But be aware that this gain often disappears after a few weeks, a classic sign of the novelty effect. Returning users in particular can be tempted to behave atypically in the short term in reaction to new elements such as a new paywall layout or different features.

To ensure you don’t fall for such test results, I recommend:

• Extending the testing time: A longer testing period helps distinguish initial excitement from real improvements. A declining difference between the test and control groups over time indicates the novelty effect.
• Analyze first-time visitors: If first-time visitors don’t show any significant differences between the test and control groups, but returning visitors do, then take results with a pinch of salt.

> You’ll also be interested in: A/B testing paywall benchmarks

2. Determine the test size in advance

It’s frustrating to discover that test and control groups were not large enough to provide valid results – especially when the test has already been completed. To prevent this, it’s crucial to calculate the ideal test size for your A/B test in advance.

The need-to-know about significance:

Not every change is an improvement. Which is why it’s important to understand statistical significance:

• Significance level: By default we use 5%, which means that we have a 5% probability of assuming an effect that doesn’t actually exist (Type I error). For smaller volumes, a level of 10% or lower may also be acceptable.

Tools like abtestguide.com allow you to check the significance of your results and experiment with different levels.

To determine the size of your test group before starting the test, Evan Miller’s sample size calculator will help you. This saves you a rude awakening later in the test, and also helps you to plan the timing of your A/B tests.

As an example, let’s assume you want to test which subscription benefits to promote on your paywall. This should increase your standard 1.5% conversion rate by at least 10%. When determining your test size, you take the following into account in Evan Miller’s calculator:

• Base rate: The standard form of the KPI to be considered. In this case 1.5% (conversion rate).
• Minimum Expected Improvement: The smallest effect you still want to see, relative to the base rate. Here, 10%.
• Alpha: The chosen significance level, 5% or 10% depending on your risk tolerance.
• Beta: The test power, usually at 80% (Beta = 0.20), indicates the probability of not detecting an effect even though one actually exists (Type II error).

For our example you would have to fill out the calculator like this:

It’s even easier with ChatGPT! Prompt for this:

My conversion rate is 1.5%. I hope my measure will result in an improvement of 10%. Tell me, according to Evan Miller, how large my test group needs to be at 5% significance.

3. One KPI per test

In every A/B test you should focus on just one KPI. Decide on a specific metric for each test, be it increasing conversion rates, improving click-through rate on the paywall, increasing dwell time, or page views per visit.

I know from personal experience that it’s tempting to look for positive signals, especially when a test doesn’t show clear results. We all desperately want our efforts to be rewarded. But be careful – the more KPIs you evaluate in an A/B test, the higher the probability of finding random positive deviations.

Let’s take our A/B testing example from above. You’re running a test to see if a new paywall layout, with different subscription benefits, will increase conversion rates. In this case, conversion rate should be your only KPI. If you also look at other metrics such as click rate or dwell time, you increase the risk of mistakenly identifying positive effects.

This risk is known as the Family Wise Error Rate (FWER). If you check the significance at 5% in an A/B test and only focus on one KPI, the risk of making a Type I error is a maximum of 5%. However, if you test seven different KPIs in the same test, the risk increases of around 30.17% that at least one of these KPIs will be incorrectly considered significant:

FWER = 1 - (1 - 0.05) ^ 7 ≈ 30.17%

This highlights the importance of focusing on just one KPI in each A/B test to ensure methodological cleanliness and statistical accuracy.

4. Alternative option

Not every publisher has the opportunity to mobilize enough users for significant A/B tests. And honestly, no one wants to run a single test for 3 months or longer. Larger publishers also face this challenge for tests further down the funnel, such as those around churn.

But before you rely completely on your gut feeling, usability tests can be a helpful alternative. They don’t provide a clear statement as to whether a measure will bring X percent more conversions or less churn, like A/B tests do. However, they do offer valuable insights to make more informed decisions.

During a usability test, you observe and analyze how users complete certain tasks. You receive feedback directly through surveys or indirectly by observing their reactions.

Let’s take our familiar example again. You guide users through different paywall versions (with and without subscriber benefits listed on the wall) and collect feedback on clarity, persuasiveness and user decisions. Typical tasks could be:

1. Evaluate paywall information
   • “Consider the information on the paywall. Please let us know what you think about the information presented and whether or not it influences your decision to proceed.”
   • Goal: Gather feedback on the clarity and persuasiveness of the information on the paywall, both with and without benefits.
2. Decision-making process
   • “Please decide whether you would click the button on the paywall to gain access. Explain your decision to us.”
   • Goal: Understand user decision-making and the factors that influence it, especially the influence of benefits.
3. Compare the variants
   • “Here are two versions of our paywall: one with benefits and one without. Please compare them and let us know which version makes you more likely to click and why.”
   • Goal: Directly compare the two versions to determine which is more effective in terms of user experience and increasing click-through rate.

5-8 conversations are often enough for meaningful results. Tools like maze and rapid user testing can be helpful for unmoderated tests.

By the way, usability tests and interviews also ideally complement A/B tests. They not only help you with optimization, but also provide information about why users behave the way they do.

I hope that these learnings are useful for your future testing projects and wish you much success in your next tests.

Speak German? Subscribe to Sascha's blog, Sub Growth or follow him on LinkedIn!  "In this blog, once a month I take a fresh and in-depth look at proven growth tactics that are specifically designed to advance the subscription business of German newspaper publishers. By sharing tried-and-tested frameworks, interesting benchmarks and exciting best cases, I want to provide practical and applicable insights that will help you in your work."

Lennart spent more than six years at the weekly newspaper DIE ZEIT before becoming a consultant last year to advise companies on how to gain and retain subscribers, develop newsletters and build communities.

The subscription economy is probably one of the most important economic trends of recent decades, which is why I talk to decision-makers from various industries about their experiences and challenges in my podcast “Subscribe Now”. The podcast is aimed at anyone who works in a subscription company and is looking for new impetus outside their direct industry. After all, many insights transfer well.

The episodes are in German, but if that doesn’t scare you off, you can find all 12 episodes of the first season here.

Here are my seven key subscription learnings from seven different industries for publishers to apply to their own strategies:

Adobe: customers first

Adobe was one of the first software companies to completely shift its business model from retail sales to software-as-a-service. What is the standard today was a bold revolution in 2013. Christoph Kull, Vice President & Managing Director Central Europe, emphasized how they have made the success of their customers their corporate goal. In the B2B sector, the special feature is that they enter into a partnership with customers. A customer pays for subscription and, as a partner, Adobe has to make sure that the promise of success materializes. That’s why SaaS companies have Customer Success Managers who don’t have sales targets, but are measured on customer satisfaction.

Bookbeat: filter out the bargain hunters

Bookbeat is one of the leading providers of online audiobooks with a subscription that offers access to over 500,000 audiobooks and e-books. Germany Managing Director, Kathrin Rüstig, explains how to build a fair subscription model for users, but also publishers. She recommends filtering out bargain hunters as early as the trial phase. Many subscription providers know the problem – users sign up for a free trial subscription and cancel on the first day. So they never intended to give the product a serious chance. BookBeat takes a special approach here because the subscription trial, which normally lasts 45 days, ends immediately upon cancellation. This allows interested users to test the app for a long time, whilst bargain hunters are quickly filtered out. This factor is all the more important since BookBeat pays a license fee for every book listened to, meaning trial subscribers also incur direct costs.

Babbel: increase CLTV

Babbel is one of the most successful language learning apps in the world and they say they have already sold 10 million subscriptions. Head of CRM Nina Pollex reveals how they build long-lasting customer relationships. In particular, they’re experimenting with subscription terms, increasing the minimum commitment from one to three months, and they haven’t noticed a significant drop in orders. Overall, they are systematically trying to make the longer subscriptions more attractive on the website in order to achieve the highest possible customer lifetime value already in the purchasing process. In CRM, they focus not only on personalization, but on individualization. Babbel tries to recommend the best individual next step to each user, because each learner has individual goals and needs. That’s why they don’t just want to create personalized communication, they want to make each user feel like they are experiencing the app in their own personal way.

Steady: community first

Membership platform Steady allows creators to be independent and fund themselves through their fans. Founder Sebastian Esser recommends focusing on your super users. Only a small part (Sebastian talks about 5%) of your users will be willing to pay for your content. If you want to be successful in the membership economy, you have to understand exactly what makes these particularly loyal ‘super fans’ tick and what will make them whip out their credit cards. But this doesn’t just apply to independent creators; established media should also try to position themselves in a specific area and not try to be all things to all people. His motto is “community first”: Most members don’t pay for content, but to feel part of a community of like-minded people. Close exchange and active community management are correspondingly important. This intensive contact can also provide valuable impetus for new products and marketing.

> To read next: From community to revenue: lessons from The Independent and Reach

Blinkist: maximum transparency

The app Blinkist summarizes more than 5,500 non-fiction books and podcasts to provides its users with an overview of the most important statements in 15 minutes. CEO and co-founder Holger Seim is particularly proud of the fact that they have created a new industry standard with their ordering process, one that is now being copied by others. The key according to Seim is maximum transparency. Even on the landing page, the steps and deadlines are listed in detail, and customers are informed before the free trial period of seven days expires that the subscription will soon cost money.

Urban Sports Club: onboarding is essential

Urban Sports Club offers a very flexible sports subscription, allowing users to train at over 10,000 partners (gyms, swimming pools or yoga studios) for a monthly fee. Torsten Müller, Senior Vice President Marketing & Communications, attaches particular importance to strong onboarding. “The first user experience should be as good as possible and happen within a certain time. Ideally in the first few days, and they create incentives for that. You should ask right at the beginning: what does the customer intend to do with the product? If you know exactly why a user subscribed to your product, then you can make personal recommendations based on that.”

Swapfiets: make customer service part of the product

Swapfiets offers bicycles on a subscription basis, promising that you will always have a functioning bike. Given this, customer service is part of the product, maybe even the most important promise of their subscription. As a subscription provider, you have to make sure that customers are satisfied with the product. To help them ensure that, Andre Illmer, General Manager for Central Europe, recommends focusing on just a few core products. Swapfiets offers only four bike models, each in different colors and sizes. With each additional product, the complexity increases exponentially, so new ideas (e-scooters, children’s bikes) are tested very thoroughly on the market, but often disregarded in favor of high customer service.