How we measure audience engagement at DER SPIEGEL

Audience engagement metrics

A method we employ for measuring audience engagement is the blending of various metrics to create a comprehensive indicator. This concept, initially popularized in the field of e-commerce, involves the renowned engagement score known as RFV (recency, frequency, volume), pioneered by the Financial Times. Using an engagement metric empowers us in two significant ways:

• Firstly, we strive to objectively measure our audience's preferences and gain a deep understanding of what drives user engagement. This entails identifying specific elements within our editorial content, product features, and user behaviors that lead to consistent engagement with our platform and offerings.

• Secondly, we use the engagement score to categorize our readers into distinct segments based on their levels of engagement. This segmentation allows us to provide personalized website interactions, such as targeted communication on pricing and content, tailored to the unique needs and interests of each reader group.

Before implementing these strategies, our approach to crafting an engagement metric consisted of two primary components: establishing a robust multidimensional methodology for measuring engagement and ensuring that engagement KPIs seamlessly aligned with business metrics, such as churn.

> You'll also enjoy: From newsletters to predicting possible cancellation: tactics to reduce churn from The Audiencers' Festival Madrid

Establish a robust multidimensional methodology for measuring engagement

While the traditional RFV factors have long been the standard for measuring user engagement, DER SPIEGEL has charted a distinctive course by introducing a novel element into the equation: Regularity. We go beyond the conventional practice of simply monitoring the frequency and volume of user visits to our site; we dig deeper, delving into the distribution of these visits over the past thirty days. In doing so, we recognize the influence of regularity in shaping user habits – a perspective that resonates with SPIEGEL's overarching business objectives, embodied in the concept of our loyalty metric called Pay Day Rate. Moreover, this approach draws inspiration from contemporary research, underscoring the pivotal role that regularity plays in the formation of user habits.

The Pay Day Rate reflects our changing strategy. At first, we were mainly focused on getting new subscribers for our subscription business. However, as time passed, we started paying attention to cancellations and the idea of “shelf life”. By 2022, we took a significant step forward by developing this new loyalty metric that has become central to our mission and revolves around assessing how many of our subscribers engage with our site consistently on a daily basis over the course of seven days. A remarkable revelation surfaced from our analysis: fifty percent of our subscribers visit our outlet every single day of the past week. This transformative shift in our approach towards user engagement reflects our dedication to high-quality content and our proactive efforts to build a loyal readership.

A further fundamental shift from the classic RFV model is the way volume is measured. Instead of focusing solely on clicks on articles, DER SPIEGEL considers usage time across the entire site. This means that user engagement is evaluated by how deeply they immerse themselves in editorial content, spanning various formats such as text, video, stories and audio consumption. Frontpage scanning also contributes significantly to regular usage and habit formation and is incorporated into the calculation.

To make all these factors (frequency, volume, regularity) comparable, they are transformed into a value range between 0 and 100. This transformation involves the use of an empirical cumulative distribution function (ECDF). ECDF transforms numerical data, indicating the proportion of values that are less than or equal to a specific value. Given that most readers tend to have relatively low frequency and volume, the ECDF transformation focuses on making changes in the lower and middle value ranges more measurable. This is a deliberate choice to prioritize the development of readers with lower to medium engagement, as increasing the engagement of highly active users is less critical. In research on media use habits, this non-linear relationship is also evidenced (“The relationship between the number of repetitions and habit strength is not linear, but with the number of executions of the media behavior, the increase of habit strength decelerates.”- from The process of forming a mobile media habit: results of a longitudinal study in a real-world setting).

Regularity is quantified by the number of unique days a user visits spiegel.de in the last thirty days. 

These values are also standardized within a range of 0 to 100 and transformed using ECDF. Recency indicates when a user was last on our site. These values are also standardized between 0 and 100 without additional transformations. Thus, a user who visited spiegel.de yesterday gets a 100, while a user who hasn't visited in more than 30 days receives a 0.

Rather than simply multiplying volume and frequency, as is done in the traditional RFV model, SPIEGEL's engagement score assigns weights to the individual factors (frequency, volume, regularity). This allows us to flexibly determine the importance of regular versus deep usage. 

Furthermore, recency remains a key factor, much like in the RFV model. Each day without user engagement diminishes the weighted score from frequency, volume, and regularity by the same proportion, underscoring the importance of keeping users consistently engaged.

Finally, every spiegel.de user, whether they have an S+ subscription or not, is assigned an engagement score based on their activity in the last thirty days. The final score falls within a range of 0 to 100, with higher scores indicating greater engagement with our site. It is updated daily and integrated into Adobe Analytics & Target, enabling segmentation, analysis, and testing. In cases where a user access spiegel.de from multiple devices and is also registered, the score is calculated across all devices, ensuring a comprehensive view of the user's engagement. The overall score is then transmitted to all end devices used by that user.

Alignment Between Engagement and Business Metrics

Boosting engagement is not an isolated mission. To ensure that we're moving in the right direction, the engagement score must align with our core business metrics. One crucial aspect of this alignment is recognizing the impact of low engagement on user retention. When users exhibit low levels of engagement, it raises a red flag. It's a warning sign that they may be on the verge of disengaging altogether, which, in turn, increases the likelihood of subscription terminations.

Looking into our data, the numbers paint a revealing picture of our subscriber base and their engagement levels: Only seven percent of our subscribers fall into the category of having an engagement score below twenty. Still, we observed a distinct trend where forty percent of cancellations were initiated by low engaged, full-paying subscribers. 

This data strongly signals the need for us to prioritize strategies aimed at elevating engagement among our subscribers, particularly those with low engagement. By doing so, we can actively work to reduce the churn rate and extend subscription duration.

On the flip side, high engagement, especially in the lead-up to and during subscription, is also a promising sign. It's laying a strong foundation for a lasting relationship. We found that users who engage deeply with our content and platform before or while subscribing, tend to exhibit an extended subscription period.

A further look at the data showed, when comparing highly engaged readers to their low engaged counterparts, we observe a distinct divergence in subscription choices: High engagement readers are more inclined to opt for long-term subscription plans, spanning three and twelve months, as opposed to monthly cancellable subscriptions (not shown in graph). This signals that readers who actively engage with our content are not only more committed but also more willing to make a longer-term commitment to our platform. Building on this insight, one of our own survey results provide further validation: Readers with high engagement express a significantly greater interest in annual subscriptions.

Conclusion

The engagement score isn't just a metric; it's a strategic tool that bridges the gap between user engagement and business success. It empowers us to make informed decisions, refine our product and content strategies, and enhance user experiences – all with the ultimate goal of not only retaining our subscribers but also attracting new ones. Our next steps involve a deeper dive into understanding the causal factors that drive user engagement and use the engagement score to categorize our readers into distinct segments based on their levels of engagement. This segmentation allows us to provide personalized on-site interactions, such as targeted communication on pricing and content, tailored to the unique needs and interests of each reader group.

Architecture (Bonus content)

All visitors to spiegel.de within the past thirty days are assigned an engagement score, regardless of whether they are subscribers. To achieve this, we need to process large-scale behavioral data from thirty to forty million users daily, demanding the deployment of parallel processing pipelines in Spark. These pipelines run inside our Azure infrastructure, where raw data is sourced from the Adobe Data Feed and stored as binary parquet files within our Azure blob storage. Our compute infrastructure relies on a mid-sized virtual machine, while Azure DevOps is used for code repositories and continuous integration. Development is done within VS Code, ensuring seamless integration with our Azure ecosystem. This project's scope includes using the engagement scores in Adobe Analytics and Target, to be able to personalize our product, conduct one-off analysis, and perform A/B testing. We used Adobe Customer Attributes, to be able to attach this high number of scores to our users daily.

Thanks to Henning Bumann, Simon Schwandner and Madeleine White for editing and proofreading.