> Audience consultant Khalil Cassimally's rule is to use synthetic audiences to narrow, never to decide. They rank options and surface objections, but cannot tell you magnitudes like willingness to pay

> Start with the decision you need to make, not the question you want to ask. Most synthetic research fails by producing interesting material nobody was ever going to act on

> Ground the panel in words real readers actually wrote. Old interview transcripts, cancellation reasons, unread survey comments. The test is whether you can name the humans inside the system

> Most teams only ever count responses. The better question is where your draft conflicts with what you have documented about a reader, which is analysis rather than impersonation

> Tell it to answer as a reader rather than an analyst, and that indifference is valid. Then validate whatever survives with real people, which is where the saving comes from

Few newsrooms are commissioning representative surveys. Meanwhile interviews cost real money and real weeks. Plenty of newsrooms also treat editorial instinct as the senior evidence, with audience feedback consulted after instinct has already decided. The effect is the same either way. When there is appetite for research, it goes to the bigger decisions while everything else gets settled without evidence.

That leaves a whole category of question permanently unanswered. The recurring ones, where we need to know:

• How a piece of content will serve the users we are trying to reach
• Which of several options is weakest, before we invest in any of them
• What objections a change is likely to run into
• How different groups of users will react to the same thing

To take one example: whether a new onboarding flow feels like a data grab. Too small to commission a study for, and it creates uncertainty that festers within the onboarding team. So somebody decides on instinct, usually whoever is most senior, and the instinct becomes the decision. It might well be right. Nobody will ever know though because nothing was tested.

This is the gap synthetic audiences fill. They don’t have to replace the research we already do well. What they change is the dozens of small calls a month that currently run on somebody’s hunch. And for the many newsrooms doing no audience research at all, the comparison isn’t between synthetic feedback and properly-designed research. It’s between synthetic feedback and nothing, and synthetic feedback done well beats nothing.

The value is that a whole class of decisions moves from opinion to evidence, at a cost and speed that makes asking routine.

What a synthetic audience is

“Synthetic audience” is an umbrella term, and the field hasn’t settled on what sits under it. You’ll see people use synthetic personas, synthetic users, digital twins, synthetic panels and silicon samples loosely and sometimes interchangeably. I’m using three here because the differences between them change what we can trust.

• A synthetic user is an LLM given a description of a type of person and asked to respond as they would. We write the description – the segment’s age range and media habits, whatever we know – put it in the prompt, and ask the question. Nothing sits underneath it. What comes back is the model’s averaged sense of what somebody matching that description would say, assembled from everything in its training data about people like that.
• A digital twin is built from one real person’s own material: an interview transcript, their survey answers, their support emails, their behaviour on our site. That material goes into the prompt and the model answers new questions as that person. It’s also possible to fine-tune a commercially available LLM on the material instead of putting it in the prompt. Academic studies from Stanford and Columbia tested that route, and neither found it beat simply placing the person’s words in the prompt. Which is good news for small teams: building a twin should not require any substantial engineering lift.
• A synthetic panel is either of the above, multiplied. We can build twenty or fifty across our segments, put the same question to all of them, and read the pattern rather than any single answer.

What synthetic audiences are good for

A number of use cases have emerged. These four have the clearest evidence behind them:

Testing language before it goes out. Paywall messages, membership appeals, onboarding copy, campaign framing. We put the draft in front of a synthetic panel and ask what they find unclear and what would make them stop reading. This is the highest-value use, partly because language is what these models handle best, and partly because it’s a decision we make most often.

Narrowing a field of options. Six newsletter concepts, or four pricing structures. The panel won’t tell us which one to launch. But it will reliably tell us which ones are weakest. So, the real research budget goes to the survivors. Spindrift Beverage Co., a US drinks brand, used a synthetic panel to choose between new product concepts. It reached the same answer in a week that a 500-person consumer study had taken two months to produce.

Surfacing objections we haven’t thought of. Ask the panel what would stop them subscribing, or what they’d assume about us from our about page, and we get a list of concerns to check. Where synthetic panels have been compared against real research, they have identified the same themes and the same objections. The magnitude and the proportions differ, the list doesn’t.

Comparing groups against the same thing. Put one paywall message to a synthetic lapsed-subscriber panel and a synthetic loyal-reader panel, then read the difference. Relative comparison is where these systems are most useful.

You may have spotted that all of the four use cases narrow or pressure-test rather than decide.

How accurate are synthetic audiences

A figure of around 85% circulates in coverage of this field, usually without much explanation. Read quickly, it suggests a synthetic version of a person gets 85 out of every 100 answers right. That isn’t what was measured.

The figure comes from the Stanford study. Researchers interviewed just over a thousand people at length. They built a digital twin of each person from their interview transcript. Then they tested whether the twin could predict that person’s answers to a survey the person had already completed.

But they didn’t score the twin’s answers against the persons’ answers and leave it at that. Because if we ask a person to fill the same survey again two weeks later, they match their own earlier answers only about eight times in ten. (Memory shifts!) So, the fair question is not whether a twin matches perfectly. It’s how a twin does against that eight-in-ten human benchmark.

Researchers found that the synthetic twins reached roughly 85% of that human benchmark – so, not 85% of the answers.

The figure tells us that synthetic twins built on rich individual material carry genuine signals about the people they came from.

The evidence in this field is still early, and most of it comes from outside media. For me, that’s another reason to use synthetic audiences for narrowing rather than deciding, which is what the rest of this piece argues for.

Where synthetic audiences stop

They can’t tell us how much. Synthetic responses cluster. Real people spread out more across opinions, while synthetic ones converge towards each other. A panel can tell us which option comes out ahead. It can’t tell us by how much, which rules out adoption rates, willingness to pay, satisfaction scores and anything else where the size of the number is the finding.

They can’t show us the small segments that matter. Clustering also means that the users at the edges disappear: the share of subscribers already on their way to cancelling, or the small core of superfans who account for a disproportionate slice of revenue. Those groups are small because they’re specific, and specificity is something a synthetic panel loses.

They tilt in a consistent direction. Synthetic respondents tend to be WEIRD (Western, Educated, Industrialised, Rich, and Democratic). They come out more comfortable with technology, less worried about privacy, better informed and more agreeable than the people they’re modelled on.

They shouldn’t speak for communities we haven’t spoken to. A team in the US building a WhatsApp media-literacy service for the Indian diaspora tried synthetic personas and got the diaspora as Westernised media imagines it. They caught it because they knew what the actual community sounds like. This is the kind of stereotype that synthetic responses can perpetuate.

How to start experimenting with synthetic audiences

Running a synthetic panel takes four steps, and each exists to make the next one worth doing.

The first fixes what you’re actually trying to settle, so you don’t generate material nobody acts on. The second decides what evidence the panel stands on, which sets how much weight its answers can carry. The third picks the kind of question you’re putting to it, which determines whether the answer is usable at all. The fourth gets an honest answer out, then records it in a form a colleague can judge for themselves rather than take on trust.

Set up once, the marginal cost of the next question is close to nothing. That’s the point. You’re building something you put questions to whenever they come up.

1. Fix the decision before you write the question

Most synthetic research fails before a panel is ever built, by producing interesting material nobody was going to act on in the first place.

Understand what you’re trying to achieve – what decision you’re hoping to make. Then determine the insights you need to inform the decision. Then figure out how to get those insights from a synthetic panel.

This ensures that the work you’re going to do is aligned with what you’re working towards. This is the step that stops the output becoming a slide nobody uses.

2. Decide what data the panel is standing on

What a panel stands on sets the ceiling on everything after it. The same question put to a well-grounded panel and an ungrounded one produces answers that may look similar but instead mean completely different things.

There are three levels:

• Nothing. A prompt describing a reader, with no evidence underneath it. What comes back is the model’s general impression of people like that.
• Demographics. Market or audience data about the segment you’re aiming at. Better, and still a description of a category rather than of anybody.
• Real material from real readers. Words that actual people you can name said or wrote.

The test is whether you can name the humans whose words are in the system. If you can’t, you’re on one of the first two levels.

Reaching the third level may be easier than it looks, depending on what your newsroom has kept. Worth looking for:

• Past interview transcripts, from any research project, however old
• Open-text answers from surveys (which usually go unread)
• Subscriber cancellation reasons
• Support and complaints emails
• Comments and community posts

Prepare the material by anonymising the person and keeping the detail that makes them specific: their circumstances, and what they came to you for. Spread it across your segments rather than piling it up. Fifteen people covering five reader types beats forty of the same type.

3. Pick the job

There’s more than one kind of question you can put to a synthetic panel. Two are worth setting out here, because most teams use the first but never the second. Choosing deliberately between them is what separates a usable answer from a merely plausible one.

• Count them. Put the same question to the whole panel and use the aggregate: which of six concepts is weakest, or which objection comes up most often. Read the ranking and ignore the margins. This is the workhorse, and it’s subject to the tilt described earlier.
• Analyse against them. Instead of asking how users would react to a membership appeal, ask where that appeal conflicts with what you’ve documented about them. The model isn’t being asked to be anyone. It compares your draft against a profile you wrote and can check, then hands back specific points of friction you can verify or throw out. Most teams never try this, and it’s the job best matched to what these systems genuinely do well, which is analyse language.

4. Make it argue back, then record what you did

Left alone, these models are agreeable, and an agreeable panel tends to tell you what you already believe.

Three things to build into the prompt:

• Tell it to answer as a reader rather than as an analyst. Otherwise the responses arrive sounding like a dull report, with tidy conclusions no human user would share.
• Tell it that being indifferent is a valid answer. Plenty of readers genuinely don’t care, and a panel where everyone holds a strong view is already wrong.
• Show it a few examples of how real people answered a similar question, with any polished phrasing taken out.

Then narrow with the panel and validate the surviving one or two options with real people. You’re only testing what has already survived something, which is where the saving comes from.

Finally, write up what you ran: who the panel was defined as, what it was grounded in, what you showed it, what you asked, what came back, and what still needs checking. Synthetic output is convincing whether or not it happens to be right. In one study, experienced researchers reviewing persona material could not reliably say which of it had been written by people. Making the output less convincing doesn’t help either, because results that read as obviously machine-made don’t get acted on. Since you can’t remove the believability, the write-up is what stops a plausible answer becoming a fact by repetition.

Start with the research you already have

The case for synthetic audiences isn’t that they closely match real readers. Academic testing has consistently found that they don’t. The case is that most audience questions in most newsrooms are currently answered by whoever is most senior in the room, and a synthetic panel grounded in things our actual readers said is better than that.

So, use them to narrow, not to decide. Ground them in real material, and know which level you’re on. Ask them to compare and to analyse. Keep them away from magnitudes, from the small segments that may matter most, and from communities you haven’t met. And, finally, validate what survives with people.

To get started: find the research your newsroom already commissioned and hasn’t opened since – the launch interviews, the subscriber survey, last year’s churn study – and work out which of this month’s decisions it could answer, if only you could ask it questions.

With a hard paywall leaving practically zero articles for free, Foreign Policy prioritizes low-effort, high-impact tactics, three of which were shared at Audiencers' Festival Montréal:
> Build one segmented "win-back" email, targeting lapsed-but-recent visitors
> Test an exit-intent registration wall instead of loosening your paywall
> Turn your best columnist into a short email course for readers who aren't ready to subscribe yet

The paywall is genuinely strict, and getting stricter

FP offers essentially no free content. Osnos was upfront that this is unusual – freemium is perhaps the most popular model in Europe, whilst in Canada a lot of content is freely accessible and many media are government supported. And their model has only gotten more restrictive over time. In 2018, the meter allowed around five free articles a month. Today it’s practically zero. Because there’s no ad revenue to fall back on, every article needs to justify its existence through subscription value.

Four problems sitting underneath that strategy

Osnos framed FP’s current position around four challenges, several of which will be familiar to most publishers reading:

• Top-of-funnel erosion. Organic discovery through Google search, historically a major channel, has dropped off significantly, even though FP isn’t chasing traffic.

• A demographic gap. FP’s core paying audience skews over 50. But it also has a large 18-to-23 segment, driven almost entirely by university group subscriptions. Those students are a meaningful exposure channel (Osnos says it’s one of the most common things she hears from subscribers she meets: “I used to read FP at university”), but they arrive through IP-authenticated access, meaning FP has no contact information for them, and loses nearly all of them the moment they graduate.

• High pricing on a strict meter. Because there’s no ad revenue, FP has to price for value and hold the line on access, which creates real friction for readers who aren’t ready to pay upfront.

• Rising competition in a moment that should otherwise favor FP. Geopolitics is dominating headlines, which sounds like a tailwind, but it also means FP is now competing directly with much larger outlets like The Washington Post and The New York Times for the same coverage. Osnos noted that FP used to see clear, isolated traffic and subscription spikes tied to major geopolitical moments (the clearest being the outbreak of the Russia-Ukraine war). More recent moments, including developments around Venezuela, have still moved the needle, but far less distinctly, because so many outlets are now covering the same stories simultaneously.

Given a small team (3 on the individual subscription side, 2 focused on group subs), Osnos’s operating principle is to prioritize low-effort, high-impact tactics, three of which she shared with us in Montréal: 

Win 1: The weekly trending digest

This is a weekly email combining a curated feed of articles, with marketing messaging at the top and bottom. Its effectiveness comes almost entirely from precise targeting and testing, not from the format itself.

Targeting: the email goes to users who visited the site in the last 30 days but haven’t visited in the last 7, a segment the team calls “Active 30, Dark 7.” These readers haven’t seen the last week’s stories, so the email surfaces what’s new rather than repeating content they may have already seen, and it re-engages them before they drift into a fully at-risk state. Readers still inside that 7-day window get more direct subscription-offer messaging instead, since they don’t need a content refresher.

Article selection: the team tested a blend of signals (last-touch conversion attribution, subscriber engagement time, and Apple News performance as a proxy for general news interest) against Sailthru’s personalization engine. Curated selection matched or outperformed the automated personalization engine consistently. Osnos’s read: curation still beats algorithmic personalization here, even though full automation would have been the lower-effort path.

Format specifics, tested directly: seven articles per edition, chosen after testing five, seven, and ten; seven won. The email always ends with an “end unit,” a final clickable item (a magazine issue, a new feature, a limited offer) so the reader has somewhere to go after the content rather than the email simply stopping. The top of the email is segmented into four different versions depending on who’s reading it, students and academics with .edu addresses see different pricing and messaging, former subscribers see a different version again, and existing subscribers never see price or offer messaging at all (they instead see messaging to reinforce sub benefits e.g. gift links, new issues, FP Live).

Effort and what’s next: the whole build takes under 30 minutes a week and consistently lands in FP’s top 10 monthly conversion sources (by campaign). The team is now testing an internally built AI tool that analyzes FP articles for conversion and subscriber-traffic performance and generates optimized headlines and decks, since the email leans almost entirely on headline and deck to pull readers back to the site.

Win 2: The exit-intent registration wall

FP has tested hard paywalls against softer meter strategies repeatedly, across every audience segment (known, unknown, newsletter subscribers, everyone), and hard paywalling consistently converts 5x better. That’s why 80% of FP’s traffic sees a hard paywall 100% of the time.

The remaining 20% is where the exit-intent wall comes in. When the system detects a reader about to leave the page (or swipe away on mobile) without paying, it offers a registration prompt instead: create a free account, get that one article. This avoids cannibalizing the primary hard-paywall conversion moment while still capturing readers who weren’t going to pay this time but are worth keeping in the funnel.

What happens after account creation matters as much as the wall itself. New registrants go through on-site onboarding and a structured nurture sequence: an explanation of why FP exists, a set of “greatest hits” articles with the paywall temporarily removed, a note from a subscriber-favorite columnist, and promotion of newsletters, which the team has found to be one of the strongest drivers of paid conversion among registered-but-not-subscribed readers, along with FP Live, FP’s video series. The average time from account creation to paid subscription across this flow is 14 days.

Win 3: Newsletter courses

FP’s first newsletter course, an introduction to core international relations theory, was launched in 2026 and authored by Stephen Walt, a Harvard professor and one of FP’s top-converting columnists (also featured in the onboarding nurture sequence above). The format is a five-part email series covering foundational IR concepts.

The logic behind it: geopolitics is increasingly relevant to people well outside FP’s usual readership, but without some grounding in the basics, a general reader has a hard time getting real value from FP’s actual coverage. The course is designed to meet a younger, less specialized audience where they are and bring them into FP’s orbit gradually.

In its first six weeks, the course drew roughly 9,500 signups. 40% of signups were net new to FP, meaning they’d never previously been in FP’s email system. Paid acquisition for the course is running at a $1 cost per lead, against FP’s normal average of roughly $3. These users are driven to the FP website rather than using an ad platform email capture, a strategy that may lead to higher CPLs but means higher quality leads.

The course also functions as a self-contained subscription funnel. Throughout its five parts, it embeds subscription messaging: “homework” assignments using gift links to relevant FP coverage, and promotion of specific benefits, like FP’s podcasts, that the team believes will resonate with a similar audience. Rather than offering a broad free trial that risks prompting cancellations among existing subscribers, FP built a narrower incentive: completing the course earns a 30-day free trial, available only to that specific, already-engaged cohort. So far, 18% of readers exposed to the trial offer go on to claim the trial.

Because the course is built once and requires no ongoing maintenance to keep running, it fits the same low-effort, high-impact principle as the other two tactics: a small team’s structural bet, built on the front end and left to keep converting on its own.

For any publisher weighing a stricter paywall against the traffic it might cost, Osnos's session is a useful reminder that the constraint didn't stop growth, it forced the team to continuously innovate and develop tactics precise enough to convert when article access can’t.

> Product decisions at most news organisations get made by whoever's loudest in the room, or by instinct built from experience. To overcome this, Germany's FUNKE Mediengruppe built a framework to replace that with something structured and evidence-led instead: a KPI tree

> Start with one metric at the top, not a dashboard. Funke picked Engaged Time, then broke it down layer by layer: what drives sessions, what drives active users, what drives new vs. returning. Each layer down gets more specific, until you hit something Stefan calls a movable metric, one a single product decision can actually shift

> Once you have even a partial tree, run a basic sensitivity check. Move two input variables by 5% each and see which one moves the KPI above it more. That tells you where to run your next sprint, not guesswork

> Only after the tree exists, build the org around it. Funke's four pillars (Reach, Engagement, Monetisation, Infrastructure) came directly from the tree's branches

> Don't start with the whole tree. Stefan's advice is to instrument one branch, show your team what it reveals, and let that build the internal case before you expand

How are product decisions actually made?

It is worth asking this question honestly, about your own organisation.

Is it the most senior person in the room? Is it instinct built from years of experience? Or is it different departments each pushing their own priorities, without a shared picture of what the product actually needs? Some news organisations have already built strong product cultures, where decisions stem from shared data and clear goals. For many others, the reality is messier.

The problem is not that any single input is wrong. The problem is the absence of a shared view, one where decisions are understood in relation to each other rather than evaluated in isolation. In practice, what often happens is that each team or department sees a part of the picture and acts on it. Everyone is doing something, but it does not click together, and a lot of value gets lost in the gaps.

When data is also missing from the equation, the risks compound. In product discussions about what to build next, maybe the loudest voice, the most confident person, or the most senior stakeholder gets their way. And maybe their suggestion is the right one. But without a shared framework to evaluate it against, there is a real chance that a lot of value gets lost. Hello, agency theory.

There is also a structural challenge specific to product development: new features are expected to have long-term impact, but evaluating that long-term impact after release is genuinely hard. Who hasn’t heard the famous “We simply have to implement Feature X, then in 6 months PV will have increased by xy%”. Too many things change between shipping a feature and having enough time to assess results. This makes it easy to keep moving by instinct, because instinct, in this environment, is rarely held to account.

So ultimately, the goal is to ground product decisions in evidence, not to take judgment out of the process.

Build the tree first

To solve this, the first thing we did was build a KPI tree.

For us as a news organisation, the metric at the top of that tree is Engaged Time, the total time users spend meaningfully interacting with content in our apps. Users choose to spend their time with us rather than somewhere else. The more intentionally they do that, the more value we deliver to them, and the more value we can generate commercially. When Engaged Time grows, the downstream indicators follow: ad impressions increase, subscription conversion improves, churn decreases.

Starting from Engaged Time, you break it down: what drives it? Session length and session frequency. What drives sessions? Active users. What drives active users? New users coming in, and existing users returning. How do new users arrive? Through different channels, direct, social, newsletter, search, each with its own conversion rate and return behaviour. How do existing users come back? Through habit, through push notifications, through content that gives them a reason to return tomorrow.

At each level of the tree, what you are really working with are ratios. How many visits convert to downloads? How many downloads become active users? How many active users encounter the paywall, and what share of those convert? Each ratio is a product decision waiting to be made. And there is something important about working with ratios rather than absolutes: it is easier to move five ratios by a small amount than to move one ratio by a large amount. The more levers you identify in the tree, the more opportunities you have to create meaningful impact.

The further down you go, the more granular the metrics become, and the closer you get to what we call movable metrics. These are metrics specific enough that a concrete product decision can actually shift them. You cannot ship a feature which directly increases Engaged Time. But you can ship a feature that improves how the homepage surfaces content, which changes the click-through rate to articles, which increases time in session, which moves Engaged Time. If you understand the chain, you know exactly what you are building toward.

Building the tree also means filling it with data. Some of it already exists. Some requires new tracking. The more granular you go, the more likely you will find instrumentation gaps. But as the picture fills in, something useful happens: you stop describing what the product produces and start understanding how it produces it.

The tree can also be filtered by user segment. The ratios look different for subscribers, registered users, and anonymous visitors. Understanding where those differences appear tells you where the most valuable product work is concentrated.

What the tree reveals

One of the most useful things a completed tree enables is simple calculation.

Once you have your first tree filled with data, even if only some branches, you can start to model the relationships between metrics. Take any KPI in the tree that is influenced by two variables below it. Increase each of those variables by five percent in isolation and see what happens to the KPI above. Which one has the bigger effect? A basic sensitivity analysis like this tells you quickly where the highest-leverage opportunities are in your product, and where effort is likely to be wasted.

Once you have identified a movable metric worth pursuing, the next question is how to actually move it. This is where mapping opportunities and solutions becomes important. For any given metric, there are usually multiple opportunities to influence it, and for each opportunity, multiple possible solutions. A structured approach, which opportunity has the highest potential, which solution is the most feasible, is what connects the diagnostic work of the KPI tree to actual product decisions. That process deserves its own deep dive, but it is the natural next step after the tree has done its job.

With that kind of visibility, prioritisation conversations change. Where should product investment go? What should marketing focus on? The tree does not make those decisions for you, but it makes the trade-offs visible. And visible trade-offs are easier to act on than invisible ones. That is the practical value: better prioritisation, and therefore better return on the investments you are already making.

A concrete example: once the tree was in place, we started observing new users as a specific segment. Sessions and page views among this group were dropping off within the first 30 days, pointing to a retention problem the aggregate numbers had been masking. Looking at what differentiated users who stayed from those who did not, the data pointed clearly to push notification subscription as a key factor. Push subscription rate during onboarding became our movable metric for this branch of the tree. We ran a series of A/B tests across the onboarding funnel, which increased push subscribers as a leading indicator, and significantly grew engaged time for this user segment, resulting in a higher 30-day retention rate as the lagging indicator.

From tree to organisation

Once the KPI tree was in place, the structure of the team followed naturally from it.

The tree has distinct branches, and those branches became four operational pillars:

Reach covers everything connected to bringing new users into the app and converting existing web readers into app users. App store presence, referral channel performance, cross-platform conversion. Reach asks: how do we grow the active user base?

Engagement takes over once a user is inside the app. Its mandate is habit formation: building the product and editorial conditions under which users return regularly and spend meaningful time. Onboarding, notification strategy, content personalisation, homepage design. Engagement asks: how do we make the app worth coming back to?

Monetisation is the commercial result of Reach and Engagement doing their jobs well. Ad revenue, subscription conversion, ARPU. Monetisation asks: how do we convert engaged attention into sustainable revenue?

Infrastructure is the supporting pillar: technical stability, data pipelines, analytics instrumentation. Without it, the other three pillars cannot move with speed or confidence. Infrastructure asks: are the foundations strong enough to build on?

And beneath all four pillars sits journalism, not as a pillar but as the foundation. Great editorial content influences all four. It drives Reach through word of mouth and search. It drives Engagement through loyalty. It drives Monetisation through willingness to pay. It makes Infrastructure worth investing in. The product framework serves the journalism, not the other way around.

The four pillars themselves are not a new idea. What matters is what sits beneath them: clear ownership, defined KPI sets, and the mandate for each team to break those KPIs further down within their branch of the tree. Downloads is a Reach KPI. But how are downloads generated? Through app store visits and the download conversion rate. Which of those can be moved? That is where the actual product work begins.

The pillars are independent in ownership but interdependent in practice. Reach without Engagement produces churn. Engagement without Monetisation produces a great product that does not sustain itself. A decision in one pillar usually affects the others.

Working together across pillars

Having clear ownership per pillar solves one problem. It creates another: how do you keep four teams, each focused on their own metrics, moving in the same direction?

This is where the organisational question becomes as important as the product question. Each pillar needs to understand what the others are working on, how prioritisation decisions are being made, and how a new feature or product change in one area will affect the others. Without that shared understanding, you end up with well-run silos rather than a well-run product.

What we built is a cross-pillar structure where all relevant stakeholders, from editorial and data to advertising, subscriptions, and tech, sit down together with the same data in front of them. Each pillar shares its learnings, its priorities, and its dependencies. Where one pillar needs something from another to move forward, that gets named and resolved together.

The goal is not process for its own sake. The goal is to get genuinely data-oriented as a group, not just in individual teams, but across the whole organisation. When everyone is looking at the same KPI tree and asking the same questions about what is moving and why, the conversation shifts. It becomes less about opinions and more about what the numbers are actually showing.

That shift is harder to achieve than it sounds. But it is where the real value of the framework sits.

A new way to think about what comes next

The KPI tree has changed how we make product decisions. The next opportunity is to apply the same thinking to strategic planning.

A user-centric planning approach starts with an understanding of how users move through the product: where they drop off, where they convert, and what drives them to return. Leading indicators first, active users, retention rates, engagement depth, push notification reach. The lagging indicators, including revenue, follow from those.

A subscriber target becomes meaningful when it is anchored to retention rate assumptions, conversion rate assumptions, and active user trends. A combination of leading and lagging indicators, connected through the KPI tree, gives every team a clear view of which part of the tree they are responsible for moving and how their work connects to the overall goal.

But the KPI tree also points toward something bigger. Every data point collected across all KPIs is a training signal. The logical next step is a statistical or AI-driven model that operates on top of the tree and surfaces the highest-leverage actions automatically. Imagine a system that tells you: sending four percent more push notifications related to sport, at 4:30pm, has a 78 percent probability of increasing MAU by two percent, which flows through to a 1.5 percent revenue effect. (Numbers are illustrative to get the point across.) The tree helps you understand your product and it can become the foundation for a system that actively tells you where to act next.

Even if we are not fully there yet, the tree gives us the language and the structure to get there.

Where to start

If you are thinking about building something similar, start with the first draft of the tree.

Start with one branch, not the whole tree. Instrument what you can, show your team what it reveals, and let the results build internal confidence in the approach. The framework earns trust the same way any good product does: by demonstrating value before asking for full commitment. Then expand.

It does not have to be complete. It will not be. Gaps in tracking will appear. Assumptions will turn out to be wrong. Metrics you thought were independent will prove to be connected. Correlation is not causation.

That is the process, not a problem with it.

The value of the tree is not in its perfection. It is in the discipline of building it, because building it forces you to ask, at every level, the question that product decisions too often skip: what needs to happen for this metric to change?

Ask that consistently. Break each number down until you find something specific enough to act on.

And when you find it, build toward that.

> When you start testing value props and benefits, it's easy to mix and match features, and try to cram everything onto one page to make subscription "worth the cost". But research has proven that the biggest driver of loyalty isn't usually price or feature count. It's a sense of uniqueness, something the reader couldn't get anywhere else

> Your team can answer this easily, but the real value comes from asking the same questions of the people you're trying to reach. At a minimum, show your prototype to one more person than has seen it already

> Once you know your differentiation, packaging decisions get easier. Rachel's 3-question test before bundling anything: does it strengthen or dilute each product's identity? Are the audiences actually converging? Can your ops hold it together without the seams showing?

> Sometimes the research tells you to do nothing. One of Skift's four products stayed untouched, on purpose, and it's now one of their steadiest, best growing subscriptions

The exercise: fill in the blank

To find that sense of uniqueness, Goligoski runs newsroom leaders through simple prompts about their organizations:

• First in the subject, region, or world to ___
• Best compared to competitors in ___
• Only organization to ___

The exercise surfaces an origin story fast. But it’s only half the picture, and it’s the easy half — anyone can answer these questions for themselves around a conference table. The real value comes from asking the same questions of the people you’re trying to reach, who often can’t name your staff or your specific value prop but can place you loosely (“oh, that’s the one about the business of fashion”). The gap between the two answer sets is the actual work. It tells you what you think differentiates you and what has actually landed.

Her rule of thumb for this kind of testing: n+1 is better than n. At a minimum, show your prototype to one more person than has seen it already. You’ll walk away smarter from even one conversation, and Goligoski says fifteen years into doing this work, people still thank her for simply being asked for their perspectives.

But watch out for “feature soup.” When you start testing value props and benefits, it’s easy to try to cram everything onto one page or into one pitch. If a first-time visitor needs a magnifying glass to understand what they’re being offered, you’ve already lost the differentiation you were trying to build.

A few examples she pointed to of publishers getting specificity right:

• Caper, a New York food media outlet, prices its annual subscription against hyper-local reference points: “less than a burrata pizza slice at L’Industrie, a potato pancake at Veselka” instead of the generic “less than a cup of coffee a day.” It grounds the price and quietly reminds readers how many small recurring charges they’re already carrying.

• The New York Review of Books and The Paris Review have a once-a-year summer bundle offer for two publications with possible audience overlap and growth potential, a financially compelling deal rather than a permanent merge.
• Capital B, a Black-media outlet in the US, sent an anniversary email that included an actual screenshot of the text exchange between its two co-founders deciding to launch. It reads as human in a way most subscription marketing doesn’t.

Goligoski’s closing asks for anyone doing this work: 

• Limit friction wherever you find it
• Take a less-is-more approach to any data you collect (only ask what you’ll actually act on)
• Keep a running, visible wishlist of the features and coverage areas you hear people ask for. That list becomes a genuine insight bank over time.

The Skift case study: is your product an audience or a feature?

Mines joined Skift, a B2B travel media and events company, two years ago and inherited what she called “a family I didn’t plan for”: four separate subscription products, none designed to sit together.

• Skift Pro: always-on travel news, home-grown, positioned as “the Bloomberg of travel”
• Skift Research: long-form, data-driven reports, also home-grown, aimed at strategy and analytics buyers making a major decision rather than staying informed daily
• Airline Weekly: a roughly 20-year-old acquired product, once faxed, now a newsletter with its own site, read closely by aviation executives (Delta CEO Ed Bastian has called it “the only thing I read every week”)
• Daily Lodging Report: a similarly old acquisition, a daily digest for hospitality and real-estate operators

Three questions before you package anything

1. Identity: does combining these products strengthen each one’s core value, or dilute it?
2. Audience: are these audiences actually growing toward each other, or are they deliberately separate?
3. Operations: can your UX, billing, and support genuinely hold a bundle together, or will the seams show? Mines runs subscriptions with a five-person team, a small product team, and a small design team, capacity is a real constraint.

Applying these questions to the Skift context

Airline Weekly: bundled the content, not the experience. Skift already had aviation coverage in both Pro and Research, plus an aviation conference. The instinct was to fold Airline Weekly’s coverage into all of it. But the products stayed on separate platforms — readers had to hop between a newsletter, a website, and (for a much smaller group) a conference ticket. Layering content across products without integrating the actual user experience just produced a disjointed one. Nobody complained loudly about it; it showed up quietly instead, in renewal numbers drifting down. 

Mines’ takeaway: audit the identity of what you’re bundling before you bundle it. Packaging without UX clarity is just confusion with a price tag.

Daily Lodging Report: the right move was nothing. This subscriber base — Mines described them candidly as mostly finance and real-estate professionals — wanted exactly one thing: the digest, in their inbox, undisturbed. Skift gave the product a new site and new data tools anyway. The response was, in effect, I don’t care, just give me the newsletter. 

So, they left it alone, and it’s remained one of their steadiest, best-growing products. Leaving a product untouched is a strategic decision, not a lack of ambition. Some products’ value comes directly from their constraints.

Pro + Research: complementary, not interchangeable. These two live in the same sales conversation but serve different budgets and different personas — daily information versus decision-support research. Combining them into one package would have devalued Research and bloated Pro. Skift’s framing: they don’t need the same package, they need the same conversation.

Mines’ shorthand for thinking about all of this is a marriage metaphor: some products should merge households (full integration makes both better); some should keep separate addresses (related, committed, but each keeps its own front door); some need couples therapy first (real potential, but only after the UX and integration work happens); and some are simply happy as they are, and the kindest, smartest move is to leave them alone.

From the Q&A: a few things worth stealing

Don’t let a new feature justify a price increase — check whether anyone asked for it first. When Skift acquired the Women Leading Travel membership group, the team raised prices and then went looking for features to justify the increase — a research report, an add-on. Mines stopped that process to ask a more basic question: did any of these members actually want the research report? They hadn’t. What they wanted was networking and job connections, and that’s where Skift redirected. Goligoski added a comparable example: ProPublica offers a branded t-shirt only after someone donates, framed as a thank-you rather than a subscription perk — about 17% of donors take it, self-selecting into people who’ll actually wear it and become visible advocates, rather than shipping merchandise nobody asked for.

Test new features free before you charge for them. Goligoski’s suggestion for the free-versus-paid feature question: launch new features free first, watch uptake, get people to describe the value in their own words, and work out the rough edges before you decide whether it becomes a paid add-on or a built-in part of the product.

When you research your gap between positioning and perceived value, don’t blend your audiences. Talk to loyal long-term subscribers, low-frequency readers, and non-subscribers separately, and be explicit — internally — about exactly who any given finding came from. Self-reported research also goes stale fast, so treat it as something to refresh regularly rather than a one-time study you cite for years.

---

The takeaway to carry into your next packaging conversation: don’t start with pricing, and don’t start with the bundle itself. Start with audience needs and wants, including the friction points people tell you about. The best package isn’t the one with the most crammed into it — it’s the one your subscriber would actually notice if it disappeared.

> stern, published by RTL Deutschland, had been built as what Chief Product & Revenue Officer Johannes Vogel calls "a portfolio machine", optimised for mass reach with the distinct identities of stern, GEO and Capital dissolved into digital scale. Workable for advertising, fatal for subscriptions

> Weekly pricing beat monthly on conversion, especially among readers with little prior attachment to the brand. The app stores went back to monthly after weekly renewal prompts created friction

> GEO and Capital now outperform on both conversion and retention, because a sharply defined brand profile answers a specific reader need rather than a general one

Lessons:
> For a general interest publisher, personalisation is not a feature — it will be the product

> The article that converts is not always the one that retains. Content consistency matters more than acquisition volume

In this guide, we share the various ways that publishers can segment their audience to develop a dynamic paywall model, with examples for each type of segmentation: 
> By content type: free vs premium, user needs, date of publication...
> By user profile: level of engagement, user status, location...
> By acquisition channel: Google, social media, newsletters...

Some quick definitions: 
> Audience segmentation: grouping users based on their profile or context, i.e. data that could impact their behaviour, inform us on what could convince them to subscribe or make them more or less likely to convert in that moment
> Dynamic paywall: a paywall that adapts for different audience segments. This could be in terms of the paywall design, messaging or even the journey itself (e.g. seeing a paywall or not). Note that "dynamic" doesn't necessarily equate to AI-driven. It's often rule based - i.e. a user reading a "politics" article sees a different paywall to a user reading a "sports" article.

Max Moné is co-founder and CEO at Poool, the dynamic journey builder to boost subscription conversion, engagement, and loyalty.

This article is the second in a 6-part series where Max shares what he learned from studying 100 subscription business models across 15+ industries. If you haven't read the first one yet, start here. 

And the other side of the coin is just as important: the DMR also showed a strong correlation between traffic on paid articles and the share of users who simply leave the site when hitting a paywall. The more you frustrate, the more people bounce. That’s not just a missed conversion. That’s a direct hit on engagement, on page views, ad revenue, and your ability to build any relationship with that user.

In the same year, Mather Economics published a study (part of a GNI LatAm Subscriptions Lab report) looking at paywall hits per user. Subscription probability follows a bell curve: up with each hit, peaks, then back down. Same conclusion, different data.

Then in 2022, the New York Times shared what brought all of this together (Audiencers analyzed it here). When they launched their paywall in 2011, the meter limit was the same for everyone: a fixed number of free articles, a regwall, some more articles, and finally a paywall.

Over the years, they developed the Dynamic Meter, a machine learning model that sets personalized meter limits per user.

The core insight: the model optimizes for two metrics simultaneously, engagement and conversion. And these two metrics have an inherent trade-off. More paywalls → more subscriptions, but less readership. They proved this with randomized control trials: as the meter limit goes up, engagement increases but conversion drops.

With a fixed approach, you’re forced to choose. Maximize engagement, or maximize conversion. Not both. There’s always a window of lost revenue that a single rule can’t capture. The Dynamic Meter expands that window by personalizing: more frustration for high-propensity users, more openness for low-propensity users.

None of this is new. DMR: 2021. Mather Economics: 2021. NYT: 2022. We’ve known this for years. And yet, the vast majority of publishers still use a uniform paywall for all users.

If you don’t personalize, you’re forced to make a choice between engagement and frustration. And that choice won’t be the right one for all users. That’s why more and more sophistication has developed. And the jump from “manual” to “1-2-1” isn’t binary. There’s a whole spectrum in between.

---

The 6 levels of paywall sophistication

Being at level 1 doesn’t mean you’re behind. But it means there are probably missed opportunities. The question is whether these missed opportunities are big enough to justify the resources needed to reach the next step.

• Level 1: Fully manual

Free and paid decided by hand, article by article. An Atlas & Audiencers’ study confirmed this is still the norm: the decision is based on the article’s perceived value. Breaking news free, analysis behind the wall (to simplify it to its core).

• Level 2: Manual + automated rules. Paid content still decided manually, but rules modify access based on additional criteria.

Foreign Policy shared their approach on The Audiencers. They segment by publish date (under 7 days: hard-walled; older: standard meter), but also by cohort. Users with high search intent and churned subscribers are hard-walled (high subscription propensity). Social referrals see a registration wall with a free page view (low subscription propensity, but conversion on registration is 100x higher than on subscription). That registration data became their strongest conversion driver. No AI needed. Just smart segmentation.

• Level 3: Manual with override.

The base decision is manual, but the system can override contextually: closing a free article for a high-propensity user, opening a premium one for a segment you want to engage. A first step toward personalization without losing editorial control.

• Level 4: Manual + a third automated category.

Free and premium decided manually, but a “grey zone” is managed automatically based on user profile, device, or source. Ideal for publishers in transition between fully manual and more automation, while still giving the editorial team a great deal of control (yes, the business side gets some autonomy, but only on categories validated with editorial).

• Level 5: Fully automated for conversion and engagement.

The system decides everything: whether to show a wall, which type, what meter limit, per user. The engagement/frustration trade-off managed dynamically.

Business Insider does this with AI: reading habits, traffic source, content genre propensity. The algorithm decides: paywall, registration wall, or nothing.

The NYT does it with the Dynamic Meter (as we talked about earlier). It learns from first-party engagement data (no demographic or psychographic features), adjusts per user, and continuously runs randomized trials to improve. This is the model that expanded the opportunity window we described above (source).

• Level 6: Fully automated for conversion, engagement, AND advertising.

The decision also integrates ad revenue. Fortune’s CCO at the time, Selma Stern, explained that if a user’s subscription probability is too low (first-time visit, smartphone, social media), the paywall opens and the page view is monetized through ads. The goal is to maximize total user value, wherever it comes from (which I personally think is an amazing model for short term profitability – because yes, sometimes long-term profitability might require less revenue in the short-term).

---

What you can take away from this (and test tomorrow)

Figure out which level you’re at. If all your users see the same paywall regardless of behavior, source, or profile: what are you missing?

Try one rule-based variation before thinking about AI. Foreign Policy’s 100x stat didn’t require machine learning. It required a hypothesis and a way to test it.

Start by looking at your bounce rate on paywalled content. If a significant share of users leave the site when they hit your paywall, that’s revenue you’re losing twice: no subscription AND no engagement. The DMR, Mather Economics, and NYT data all showed the same thing: a uniform paywall always forces a trade-off. Even a simple first step (like showing a registration wall instead of a paywall to low-propensity users) can start closing that gap. That’s what Poool makes it easy to test.

See you next week for Article 3, where we leave media and look at how the best subscription businesses across all industries think about conversion.

---

PS: This article was about the decision logic (who sees a paywall, and when). We didn’t cover the blocking method: how the paywall is technically implemented, front-end or server-side, and what that means for content protection, AI scraping, and SEO. That’s a separate topic, and we covered it here: “Paywalls & SEO“. All 6 levels above work with all blocking methods.

Khalil A. Cassimally is an audience consultant and coach who helps organisations align teams around change, user needs and AI. 

Audience and funnel work succeeds or fails on internal communication, yet few people get structured support to improve it. Drawing on recent research and his own practice, he shares a practical three-step way to use AI to:
> Understand your intervention style
> Identify strengths and gaps in how you communicate
> Take small, intentional steps to improve how your communication skills