Your Synthetic Audience Looks and Sounds Like Your Consumer. That Doesn’t Mean It Thinks Like One.

Michael Maslansky

CEO, New York City

Digital twins and AI personas are everywhere right now. When it comes to testing your message, they’re probably giving you a false sense of confidence.

So, you’ve built a digital twin to represent a key audience segment, perhaps frequent fast-food consumers , DC policy leaders, or oncologists. To make it accurate, you layered in demographic data (age, income, race, location), psychographic insights (values, beliefs, attitudes), and behavioral patterns (purchase frequency, voting behavior, prescribing preferences). Using this approach, you’ve built a “typical” synthetic persona that sounds uncannily real when you ask it a question. The responses are specific, nuanced, and almost scary in their authenticity.

But are they accurate?

Based on our research, the answer is probably not.

How do we know? We tested it, across multiple audiences and more than 100 messages and lexicon data points. In each study, we built a “typical” synthetic persona for a specific audience. We then tested the answers provided against human responses from the same audience to the same questions about the same messages. The result was not encouraging. Built with demographic, psychographic, and behavioral data alone, these personas only got the answer right between 30% and 50% of the time. Not much better than a coin flip.

What’s in a message?

Over the past two decades, my firm has conducted thousands of studies with hundreds of thousands of respondents. As a firm that specializes in Language Strategy, our singular focus is the study of effective messaging—what works, what doesn’t, and why.

For the past two years, we’ve worked to develop a validated approach to conducting synthetic research. We’ve learned a lot along the way. But perhaps the most important lesson is that more data does not lead to better results, and just because your synthetic audience looks and sounds like the real thing, it doesn’t mean they can help you build a better message.

Across our tests, synthetic personas could get the big things right. They could identify which messages are relevant, which are easier to understand, and which appealed to specific aspects of their persona (gender, age, etc.).

But people are highly sensitive to nuanced differences in messaging. And synthetic personas built solely from demographic, psychographic, and behavioral data (we’ll call them “typical”) consistently fail to predict how subtle message enhancers and detractors drive persuasiveness with human audiences.

In one study, our food industry client wanted to identify the most effective way to communicate all the steps it was taking around nutrition to an audience of policy leaders. The answers from our typical synthetic policy leader audience made intuitive sense. But they were wrong.

• When evaluating the strength of food-related ingredient claims, typical synthetic policy leaders overvalued stakeholder-focused language (“responsibly sourced ingredients from sustainably managed fisheries”). Actual policy leaders respond more favorably to concrete and provable language (“simple ingredients, like 100% pure beef”).

• When asked to identify which approach from a company would best help customers make informed choices, synthetic policy leaders overweighted policy-coded language (“transparent nutritional information”). Real policy leaders prioritized optionality (“wide range of options”).

These simple examples reflected systemic biases in the personas that we could see repeated across studies with the same audiences.

Receptographics: The Missing Variable

While more—or more selective—use of demographic, psychographic, and behavioral data failed to improve predictive abilities, another category of data had a dramatic effect.

Using our body of narratives, messages, and words and phrases, all tested in qualitative and quantitative research with human respondents, we identified patterns that consistently influence how different audiences respond to messaging. When integrated into our personas, the predictive power improved from 30-50% to 80-90%.

The missing ingredient is what we call “receptographics.” Just as demographics define who the audience is, and psychographics define what the audience believes, receptographics define how that audience responds to different information cues embedded in every message.

The distinction is critical. Demographics, psychographics, and behavioral data can be effective in helping the LLMs predict which issues matter and which policies are preferred when the choices are clear and distinct. But the LLMs have not yet learned how to use this data to make sense of nuanced differences that are central to effective messaging (e.g. framing, tone, word choice). For that, we need receptographics.

Understanding policy leader receptographics

Let’s go back to our policy-leader example. These regulatory officials, legislative staff, institutional decision-makers, and advocacy leaders are among the most demanding message audiences our clients encounter. They’re skeptical by training, sophisticated evaluators of evidence, and politically attuned to language that is often subtle but highly coded. While they can have wildly different views about a topic or a policy, they are remarkably consistent in how they respond to different approaches to messaging.

Here are some examples: 

These aren’t individual examples. Each is a repeated pattern of response that exists whether we are testing food claims, autonomous vehicles, or data privacy.

Finding the right tool for your challenge

Most synthetic personas are built to model who an audience is, what they believe, and how they behave. But message performance depends on how an audience interprets what it hears. That is a different problem, and one most LLMs—and the synthetic personas built from them—are not reliably trained to solve. 

We built CHORUS, maslansky + partners’ proprietary approach to synthetic research, to specifically address this challenge. CHORUS audiences are trained with audience-specific data about how their human analogs respond to messages. CHORUS also uses the same research methodology we have used for thousands of human tests. Our approach results in higher predictive accuracy and the ability to compare effectiveness against benchmarks for studies with similar objectives. 

The biggest risk associated with synthetic audiences is that they all sound quite persuasive. Their answers make sense. But that doesn’t mean they are right and they could be wrong more than 50% of the time.  To get more confidence than a coin flip, contact us.

You Might Also Like