A voter surveyed in October may give a different answer by January. An AI persona might not, depending on how it’s built.

That gap is one reason Jennifer Stromer-Galley, professor in the School of Information Studies, is cautious about synthetic polling, where polling firms will often build hundreds or thousands of personas on large language models (LLMs) based on existing polling data, along with other demographic information to create stand-ins for real voters in focus groups and surveys.

“We know from research that if you survey somebody at the beginning of October on what they feel are the most important issues facing the country, and then you survey them again three months later, they might give you different answers because their circumstances have changed,” Stromer-Galley says. “My concern is how the chatbots account for shifts in public opinion.”

Historically, public opinion polling has been valued for its reliability, predictability and consistency. Synthetic polling clouds that picture.

AI personas rely on background and key demographic information that is relevant to the poll when generating their collection of prospective voters. LLMs can be backfilled with materials or documents to further round out the personas, creating a more detailed representation of a voter and their potential positions on a policy or a range of policies.

Down the road, polling vendors could devise artificial personas based on detailed policy stances and perspectives to run artificial focus groups and have them participate in interest-based surveys.

“We’re at the beginning of the synthetic persona era,” says Stromer-Galley, who has studied the methodology involved in public opinion polling. “Polling firms see this as a great opportunity to potentially expand their offerings at a relatively low cost, and because they don’t have to deal with the logistics and cost of getting people to answer their surveys.”

Personas May Miss Changing Perspectives

While Stromer-Galley says LLMs will consistently rely on the instructions and information provided when building those personas, that consistency could lead to polling groups that don’t accurately reflect the views of the intended audience.

Traditional polling captures shifts in opinion because it surveys real people each time. Synthetic personas are built on fixed instructions, so their answers tend to stay consistent from one round to the next.

Actual voters, shaped by changing circumstances, can and do change their minds on the issues.

“Humans don’t have internally coherent policy frameworks; it’s much more complex how people think about policy matters,” Stromer-Galley says. “These personas will give you consistent answers every month. There could be some variance, but that variance range would be much narrower than with people.”

Accounting for Outliers

Stromer-Galley says a good, nationally representative public opinion survey should include roughly 1,200 individuals. Synthetic polling firms, by contrast, can generate several thousand or more personas. The bigger concern is not sampling noise but bias within the AI models that drive those personas. Vendors can adjust the demographic mix  of their personas, but that doesn’t correct for how the model answers for each profile.

Consider a hypothetical survey of rural Black voters in a small town. For a group like this, Stromer-Galley says the AI models and synthetic data will give typical answers for a profile, which means actual human outliers can disappear in the synthetic approach.

But only adjusting for demographics a persona is built on, Stromer-Galley says, can’t show the range of views real people hold.

“Just because the number of personas matches the average demographics of the population, you’re still going to deal with outliers in the survey responses,” Stromer-Galley says. “How will synthetic polling build in for and account for those outliers, and how would they know that’s an outlier?”

A Pew Research Center experiment published in September suggests the concern is more than theoretical. Pew had an AI model take on the personas of real members of its American Trends Panel and answer the same surveys the human panelists took. Across nearly 300 questions, the AI results differed from the human results by an average of 12 percentage points, and nearly half the questions had at least one answer choice that no AI respondent selected.

‘Polling Has Always Evolved’

When scientific public opinion polling first occurred in the 1930s and 1940s, part of its allure was “it allowed a nation to understand its concerns in ways that fostered this greater sense of belonging to a collective,” Stromer-Galley says.

While political conversations with neighbors and friends were common, there was something unique in learning how the rest of the country felt about the candidates and the issues in an election.

Public opinion data is still being used to understand the country’s concerns, values and policy positions. But compared to its origins, Stromer-Galley says synthetic polling is a “distorted mirror of us that potentially creates misunderstandings that get magnified.”

“The AI data is us, but it’s also not us. It is a reflection of us, but it’s also refracting us in ways that aren’t quite representative of who we are,” Stromer-Galley says. “Once we start replacing actual human voters with synthetic voters, how does that change our sense of our reality about ourselves? If we’re relegating those insights about us into synthetic data, what do we lose?”