This article was originally published by Frank Bi on Hacks/Hackers, republished with permission.

> Journalism has to be right the first time. Product work learns more from what doesn't work. Frank Bi, AI Strategy and Product Advisor at Hacks/Hackers, argues that when products are held to the journalistic bar, nothing half-baked ever launches, so nothing ever teaches you anything

> Build a public sandbox. A labelled subdomain (beta, labs) resets the expectations readers bring to the homepage while still producing real behaviour data. Bi launched the Minnesota Star Tribune's AI restaurant guide Culinary Compass on the main domain instead, where readers judged a four-week experiment as a Star Tribune guide

> Set the end date before launch, ideally by picking something with a natural start & end. Bi's Fair Bot ran only the 12 days of the Minnesota State Fair, so nobody ever had to make the call to shut it down

> Show your work. Newsrooms already have a practice for issuing corrections, so extend a version of it to experimental products and publish what the team believed and why, before that thinking disappears at launch

There’s a tension between journalism and product innovation, and it comes down to a clash of mental models. Journalism requires getting the facts right the first time. On the other hand, product innovation learns more from what doesn’t work than from what does, and sometimes that means putting something half-baked in the wild to find out what audiences will actually respond to.

Luke Williams, a former creative director at frog design who taught my innovation and design class at NYU Stern this spring, has a line for this: be wrong at the start to be right at the end. It resonated because I’ve been in news organizations where the journalistic mental model is applied, often for good reasons. But when the product is held to the journalistic bar, nothing half-baked gets launched to teach us anything.

The post How newsrooms can learn from failure without failing in public appeared first on Audiencers.

Reuters’ Elaine Piniat and Judith Langowski share how they’re using A/B testing to optimize newsletter performance without getting lost in complex tweaks.

> Reuters found that stripping things out beats adding them. Removing images from list-style newsletters lifted click-through by 29 to 33%, and removing item descriptions lifted it by an average of 59%

> That descriptions test came from a Washington Post colleague at a conference and was confirmed by former Newsday staff. Reuters ran it across three newsletters rather than assuming it would transfer

> Moving the newsletter sign-up module to the top of articles cost some ad revenue. Testing showed it performed 50% better, so the trade was justified. Module clicks then rose 63%

> Position kept mattering more than content. Click-focused sections placed higher in narrative newsletters always won, and an image moved from top to bottom lifted click-through by 15%

> For the Daily Briefing relaunch the team built three prototypes and surveyed 300 readers. The shortest, most concise version won over versions offering background or a deep dive

A cornerstone of audience strategy is a/b testing. The media industry, which is frequently inundated by new and shiny tools and ways of telling stories, must constantly iterate to keep our audiences engaged and informed. It’s easy to get lost in the competition and tweaks to algorithms, leading to overthinking and overdeveloping. But a/b testing doesn’t have to be so complicated. If we keep the reader at the center and go back to the basics, we can see meaningful impact.

The post 5 essentials when A/B testing from Reuters’ newsletter team appeared first on Audiencers.

FAZ's Donika Lilova spoke at The Audiencers' Festival London in June 2024 on their digital subscription journey and, in particular, about how they're working on making their paywall dynamic with regular A/B testing to continuously optimize performance. In this article Donika dives deeper into this strategy.

Hi! I’m Donika and Head of Paid Content and Deputy Chief Subscription at Frankfurter Allgemeine Zeitung (FAZ). My responsibilities include the management of the Paywall Team.

Digital growth is a core objective of our company strategy, and we are on track to reach our subscription goal of 300,000 paying digital subscribers by 2025, in part thanks to our A/B testing and dynamic paywall.

The post Dynamic paywall testing: how FAZ is working to reach 300,000 digital subscribers by 2025 appeared first on Audiencers.

A/B testing is essential to being successful in the subscription business. Whether Spotify, Netflix, Disney or The New York Times - everything is tested before it is rolled out on a large scale.

I have conducted over 100 A/B tests in the last few years and would like to share my top 4 learnings. Although these insights come from a non-data perspective, they are particularly valuable for anyone approaching the topic from a product or growth perspective. My learnings should serve as an impulse for you to delve deeper if you are interested.

The post 4 learnings about subscription A/B testing from ZEIT ONLINE appeared first on Audiencers.

"Making decisions is easy, what’s hard is making the right decisions."

The post Constantly evolving product offerings with A/B testing at Netflix appeared first on Audiencers.

When it comes to continuously optimizing performance, there's nothing more valuable than A/B testing. But EBRA media, France's biggest regional press group, have gone a step further to develop a culture of experimentation across their 9 brands, testing out user experiences, journeys and paywall design to maximize conversion rates. 

Goal: Increase the number of digital subscribers across EBRA’s 9 brands 

Solution: Continuously test to validate hypotheses, analyze results and apply these learnings to their reader revenue strategies, discovering exactly what factors are needed to successfully convert users into subscribers. 

Examples of experimentation at EBRA: 

• Soft conversion steps
• Hybrid paywall
• Temporary wall designs
• A/B testing 
• Subscription offers in the paywall

Soft conversion steps

Goal: increase engagement prior to the paywall to boost conversion rates

Solution: test out soft conversion steps

• Data wall
• Registration wall

A Poool study recently found that, on average, over 80% of a publisher’s audience are volatile, unengaged users who only visit about once a month. 

So, whilst a hard paywall might convert a small percentage of very engaged fan users, the large majority of audiences are going to be frustrated by the paywall, perhaps even leaving the site to find content elsewhere. 

To solve this, publishers should employ a dynamic strategy whereby conversion journeys are adapted to the reader’s profile or context with the goal of increasing engagement and maximizing ARPU. 

EBRA understood this and decided to try out a data wall, offering users access to the premium article in exchange for their email address as a key data point for marketing campaigns.

The team also tested a registration strategy, asking users to create a free account in exchange for access to premium content for 24 hours. 

What happened after these soft conversion steps? 

Adapted journeys were built for users who passed through these soft conversion walls, presenting paywalls with personalized messaging. 

Results: these strategies were hugely valuable for soft conversion uptake and building a strong foundation of an engaged audience – in fact, Autumn 2021 saw a large increase in users in the engaged segments: +236% occasional readers, +182% regulars and +98% fan users.

Hybrid paywall

Goal: increase conversion rates and engagement simultaneously

Solution: combine soft conversion with a hard paywall to increase the visibility of their premium offer whilst also forming a habit through the newsletter.

Top half of the wall: newsletter sign-up with a select field to choose which newsletter to sign up to

Bottom half of the wall: “Fancy discovering the subscription experience? Sign up from $1 for the first month, cancel anytime.” – promoting subscription to either convert or educate users on the idea of paying for access to quality content

Temporary walls

Goal: Reduce habituation and avoid ‘paywall blindness’ to reignite conversion rates and maximize on special offers

Solution: build walls for different national holidays, French elections or sales periods, analyzing whether this has an impact on click-through and conversion rates. 

Results: This Christmas paywall led to an increase in click-through rates across all audience segments: 

• +58% globally
• +75% volatiles 
• +146% occasionals 
• +149% regulars 
• +65% fans

Best practice: analyze results globally but also for different audience segments as data will likely differ, giving you insights into how you can better personalize the reader journey to their profile or context. 

A/B testing

Goal: continuously optimize click-through and conversion rates

Solution: the EBRA team carried out a variety of tests on paywall design and wording with Anthony Ribeiro, Poool’s Consultancy Lead. 

For instance, they tested out a few value proposition texts on their paywall:

• Version A – shorter text where the focus is on the publisher accompanying its reader in their daily habit of taking a morning coffee whilst reading the news. 
• Version B – focus is on the benefits of subscription, such as unlimited access and the cost of their offer. 

Results: Interestingly, paywall A achieved 2 times more clicks globally than B, suggesting that shorter text and framing messaging as ‘accompanying’ the user’s existing habit is beneficial for CTRs in this context. 

“It’s essential to maximize A/B testing to find the best strategy, the right messaging and the perfect offer to display to our audience. Poool ‘s solution allows us to set up all the engagement and conversion scenarios we can imagine with our teams, in complete flexibility and autonomy. This allows us to better understand our audience and personalize messaging as much as possible with the goal of maximizing conversion rates.” 

Emilie Courmont, Acquisition Project Manager

Integrating subscription offers into the wall 

Goal: increase conversion rates by reducing friction in the funnel

Solution: remove a step in the conversion funnel, with the CTA button leading straight to payment 

Results: after testing this wall on one of their brands, the results were clear:  click-through rates increased by 97% and conversion rates by 37%, leading to the group deploying this offer wall across all 9 of their brands. 

Best practice: don’t forget to build a mobile-friendly wall to improve the user experience on smaller screens. EBRA, for instance, only features one offer on the wall on mobile devices. 

The Audiencers’ newsletter: from professionals to professionals

Sign up to our newsletter – real-life examples, expert points of view and inspirations from publishers around the world to help you do your job better. Sent every two weeks.

The post Developing a culture of experimentation at EBRA Media appeared first on Audiencers.

Optimizing your paywall and subscription strategy to compete for a place in your reader’s monthly budget is no mean feat. And whilst there are endless benchmarks and success cases online, there is really only one way of finding the best performing strategy for your unique audience: testing.

Being one of the most user-friendly but efficient ways of optimizing your paywall performance, testing is something that is highly recommended for publishers to do on a regular basis and in a variety of forms.

But, unlike many of the other optimization methods, it is not the easiest to benchmark given that the whole point of running an A/B test is that users are unaware when one is taking place.

Here are three examples of paywall A/B tests run by digital publishers – what can you test, what do you need to define prior to launching the test, what were the results and, most importantly, what can we learn from these to apply to your own tests?

Bio à la Une

This French health and well-being publisher, who only recently launched a premium strategy, wanted to learn more about their audience and what makes them convert. They, therefore, ran a series of tests, one of these varying the wording on the call-to-action button.

Hypothesis: Click-through rates (CTRs) will increase by using a more direct call-to-action phrase, “je m’abonne!” (Subscribe), one tha is commonly employed by digital publishers in France.

Variable: Text on the CTA (call to action) button.

Goal: Increase CTRs and discover the focus that works best for each audience segment.

Control (version A): CTA button text “Discover our offers”, a more invitational, indirect phrase.

Treatment (version B): CTA button text “Subscribe” takes a very direct approach.

Results: Version B had a higher click-through rate amongst less engaged users, which suggests that these audiences respond better to a direct CTA phrase. Version A, which invites readers to discover subscription offers, performed better for more engaged users.

This was an interesting result, and one that highlights the importance of segmenting your audience before testing. In this instance, the publisher used Poool’s Dashboard to segment users based on level of engagement (Volatiles, Occasionals, Regulars and Fans) which gave insights into how these users respond differently to direct vs indirect wording on the paywall.

To further optimize your paywall performance and gain a deeper understanding of user behavior, you could also consider segmenting audiences based on location, device or even a users interests if you’re able to gather this information.

The takeaway: Just as the performance of a paywall will differ on your site compared to that of another publisher, this can also be said for one segment of your audience compared to another.

ELLE Magazine

The fashion and beauty magazine launched its digital premium strategy with a two-step conversion journey consisting of a registration wall followed by a hard paywall. This proved hugely valuable for increasing engagement and informing their audience of the concept of paying for access to content, but the team wanted to test a more direct approach to see if this led to higher conversion rates.

Hypothesis: A hard paywall will lead to higher conversion rates compared to a soft engagement journey.

Variable: User journey.

Goal: Increase conversion rates.

Control (version A): A two-step scenario with a soft conversion registration wall followed by a paywall.

Treatment (version B): A hard paywall alone.

Results: Version B led to slightly higher conversion rates than version A. However, these results have to be taken with a pinch of salt…

Firstly, it is important to consider the stage of your subscription model. For ELLE, although version A worked extremely well for launching its strategy, collecting data and increasing engagement prior to subscription, version B performed better when this test took place, meaning their audience was perhaps more qualified and less likely to be frustrated by a hard paywall.

You also have to consider more than simply conversion rates as a KPI (key performance indicator) when altering your user’s journey, namely ARPU (average revenue per user). Considering engagement has a direct correlation with revenue, and that soft conversion steps such as registration walls increase user engagement, it is arguably more beneficial in the long term to employ a user journey such as version A, despite what these test results suggest.

Specifically, de-anonymizing a user through registration increases ad-based revenue thanks to targeting and also boosts propensity to subscribe (and increases the likelihood that this user will remain subscribed in the long run).

The takeaway: When looking at benchmarks, do not forget to consider the context surrounding them, take overall ARPU into account and remember that it is not only the paywall that influences conversion rates.

For instance, although ELLE did choose to employ journey B on all user segments, it also worked on increasing the visibility of premium content, reducing frustration by adding yellow tags on these subscriber-only articles as well as boosting engagement in other areas, such as by promoting their newsletter and membership offers. These efforts had a big impact on balancing frustration and engagement which ultimately support high conversion rates.

Neos Kosmos

One of the most important aspects of your premium model is the value proposition. Why should a user subscribe? What pain points will it solve? How will it help the reader achieve their ‘jobs-to-be-done’?

And, of course, this value proposition needs to be clearly communicated to readers on your paywall in a way that is easily understood and gives enough encouragement to click-through and subscribe.

This is something that the Australian-Greek publisher Neos Kosmos understands well, leading them to test out a shorter version of their value proposition which focuses more on ‘supporting’ the publisher than on the benefits of subscribing.

Hypothesis: A value proposition focusing on ‘supporting’ Neos Kosmos will lead to higher conversion rates.

Variable: Wording on the wall (value proposition).

Goal: Increase conversion rates.

Control (version A): The original wording details what Neos Kosmos’ premium offer gives a subscriber access to.

Treatment (version B): This version takes a more ‘support our work’ approach, appealing to a user’s emotional and cultural attachment to Greek culture in Australia.

Result: Version B led to a 16 per cent increase in conversion rates.

For Neos Kosmos, this showed the importance of appealing to their audiences’ cultural heritage and the shared values amongst readers. For many publishers, taking the angle of asking for support rather than providing access to content is hugely valuable for increasing conversion rates and developing a community of loyal subscribers.

The takeaway: Value propositions need to be clearly defined prior to launching your premium strategy but should also be tested on a frequent basis to discover the optimal angle, wording, etc. for your unique audience.

To summarize:

• Clearly define your hypothesis, variable, goal and audience segments prior to launching the test
• Remember that there is more at play than just the paywall, such as value proposition, visibility of premium content, the balance between frustration and engagement, etc.
• Tracking click-through and conversion rates alone has its limitations – ARPU is arguably the most important
• As always, benchmarks are great and useful for inspiration, but there is no one-size-fits-all strategy that will work for every digital publisher

Happy testing!

The post How publishers are using A/B testing to optimize conversion rates appeared first on Audiencers.

A/B testing is the number 1 strategy to optimize your paywall click-through and conversion rates, allowing you to discover what performs best for your unique audience, content and business model.

Given its value, we’re here to share benchmark examples and insights from some digital publishers carrying out A/B tests.

Note: it’s important to remember that every publisher is different, meaning test variables and results may not be the same for your strategy. Having said that, it’s useful to consider what others in the industry are testing to take inspiration for your own tests!

The goal: Increase click-through and conversion rates!

Quickly, what is A/B testing? 

A/B testing, or split testing, is a marketing technique that works by comparing the performance of two versions of a single variable (such as scenarios, wall layout, colors, copy) to discover which leads to the highest conversion rates. The variant being tested is called the ‘control’ whilst the variant that is argued to give a better result is called the ‘treatment’. 

France Antilles

Hypothesis to test: Does the length of the value proposition and text on a paywall influence CTRs? 

Variable: Wording (value proposition length)

Goal: Increase click-through rates across all audience segments

Control (version A): Detailed value proposition detailing the key benefits of subscribing in 5 bullet points. The wall also had two CTA buttons, one for the average individual reader (‘Try our digital offer at 1 euro for 30 days’), the other for group subscription offers (‘Request a personalized offer’). 

Treatment (version B): Here we have a simple single sentence value proposition aiming to create a community around shared cultural values with just one CTA button (‘Continue reading’) which moves the focus away from payment. 

Result: Version B, the wall with a shorter but more personalized value proposition, performed 5x better on average than version A, and this was the case across all audience segments and websites (the publisher has 3 in total). 

Top tip: we wouldn’t recommend modifying multiple aspects of the wall in one go (e.g. the CTA button text and value proposition text) as this takes the focus away from analyzing and optimizing a single feature in one go. 

Le Journal du Dimanche 

Hypothesis to test: Does the color of the JDD Newsletter Wall influence CTRs on mobile devices? 

Variable: Color of the wall

Goal: Increase CTRs on mobile devices (where an increasing percentage of users access their site but where conversion rates still remain low) 

Control (version A): Gray wall with yellow CTA button

Treatment (version B): The inverse, so yellow wall with gray CTA button 

Results: The yellow wall far out performed the gray, persuading the team to employ version B over the other. 

Bio à la Une 

Hypothesis to test: Does the text on the call-to-action button influence CTRs?

Variable: Text on the CTA button

Goal: Increase CTRs and discover the focus that works best for each audience segment

Control (version A): CTA button text ‘Discover our offers’

Treatment (version B): CTA button text ‘Subscribe’ (note that ‘Je m’abonne’ has become a common phrase in France associated with subscriptions for digital publishers)

Results: Version B has a higher click-through rate amongst less engaged users, whilst version A performed better for fans who are highly more engaged.

Top tip – segment your audiences and run your tests on each individual group because you may well get different results based on the ‘type’ of reader. 

For instance, you could split users based on the device used, engagement level, location or whether they’re anonymous or known visitors. 

ELLE Magazine

Hypothesis to test: Does the wall format make users more likely to click-through the paywall?

Variable: Paywall format

Goal: Increase click-through and conversion rates for all user segments

Control (version A): The standard 

Treatment (version B): A banner version 

Results: The original format worked slightly better amongst volatile (less engaged) users whilst the banner version significantly improved conversion rates amongst occasional readers.

Neos Kosmos

Hypothesis to test: Does a different tone of voice on the paywall influence conversion rates?

Variable: Wording on the wall

Goal: Increase conversion rates

Control (version A): The original wording details what Neos Kosmos’ premium offer gives a subscriber access to

Treatment (version B): This version takes a more ‘support our work’ approach, appealing to a user’s emotional and cultural attachment to Greek culture in Australia

Result: 16% increase in conversion rates with version B

Second Neos Kosmos test

Hypothesis to test: Whether the visual employed on the paywall affects conversion rates

Variable: Visual

Goal: increase conversion rates

Control (version A): The publisher actually moved from the version above (with no visual) to one with a visual, testing which version led to higher conversion rates. Version A was an image with immigrants arriving on a boat

Treatment (version B): This version is of a Greek building

Result: 13% higher conversion rates on version A with the boat

Top tip – Do as Neos Kosmos has done – A/B test continuously! You may increase conversion rates with one test but who’s to say that another version couldn’t increase them even more?

Quotidien & Tageblatt

Hypothesis to test: Whether CTRs increase when messaging is altered for a special event/national holiday

Variable: Text

Goal: Increase click-through rates

Control (version A): The original wall

Treatment (version B): A special version of the wall for Press Freedom Day. This wall was presented to readers over a 5 day period from April 30th-May 4th on both Quotidien and Tageblatt

Results: The results are very positive. There was a clear increase in click-through rates on the paywall, showing the importance of a contextualized message that’s aligns with the media’s identity.

For Le Quotidien, CTRs increased by up to 18%, whilst Tageblatt saw an increase of up to 4%.

Le Petit Journal

Hypothesis to test: Does the length of the value proposition influence CTR AND does the result differ based on the type of content?

Variable: Wording

Goal: Increase click-through rates

Control (version A): The original wall with value proposition points and a focus on supporting the publisher (i.e. subscribe to support us)

Treatment (version B): A shortened version without the bullet points and a focus on the fact that the user hasn’t finished reading (i.e. subscribe to continue)

On premium content –

Results: Version B had 23% higher click-through rates

On free content –

Results: Version B had 59% higher click-through rates

The Audiencers’ newsletter: from professionals to professionals

Sign up to our newsletter – real-life examples, expert points of view and inspirations from publishers around the world to help you do your job better. Sent every two weeks.

The post A/B testing paywall benchmarks appeared first on Audiencers.