Skip to main content
Joshua Thompson

AI

Colgate’s AI consumers can answer a survey. They cannot buy the soap.

A study of 57 personal-care concepts shows where simulated shoppers could save a product team time, and where it still needs people.

By Joshua Thompson

5 min read

An unbranded soap bottle and an orange in a wire basket, surrounded by translucent human-shaped cut-outs.

A customer who loves your new product is encouraging. A customer you invented needs a different kind of scrutiny.

The appeal of AI-generated consumer panels is easy to understand. A business could test more ideas, explore objections and improve a weak proposition before paying for a large research exercise. The danger is just as obvious: a persuasive conversation with a simulated buyer can feel like demand when nobody has decided to spend any money.

That distinction runs through the Colgate-Palmolive and PyMC Labs research. The October 2025 preprint is an interesting attempt to reproduce consumer survey results. It is not evidence that an AI panel can guarantee a product launch, and it does not show that Colgate replaced 9,300 customers with chatbots.

My view is that this technology could be most useful before a business has a polished product to defend. Used to challenge ideas cheaply, it has a clear role. Used to supply reassuring numbers for a decision already made, it could make bad research considerably faster.

What the Colgate study actually measured

The researchers compared synthetic responses with 57 personal-care concept surveys involving 9,300 US respondents. Rather than simply ask a model for a score out of five, their semantic similarity rating method turns free-text reactions into rating distributions by comparing them with reference statements.

For each concept, the team compared the average score from its simulated shoppers with the average from human respondents. The two sets of scores tracked each other about 90% as closely as two halves of the human panel did. That benchmark came from repeatedly splitting the human samples. It does not mean that 90% of individual answers were correct, or that nine out of ten forecast purchases occurred.

That is a substantial distinction. A tool that helps sort promising concepts from weaker ones may be commercially valuable. It still has to earn the right to predict how many units a retailer should put on a shelf.

In the webinar below, PyMC Labs explains the scoring method and introduces its own synthetic-consumer platform.

Video not loading? Watch on YouTube

Saying yes and paying are different events

Imagine a proposed refillable hand-soap bottle. A simulated respondent says the packaging looks attractive, the environmental benefit appeals and the price seems reasonable. That is a useful starting point for a research question.

Now put the bottle in a shop. The shopper may be carrying a child, buying something else, facing an empty bank account or reaching automatically for a familiar brand. The refill may be out of stock. The pump may be awkward. A discount on another product may settle the decision.

These are ordinary reasons why a plausible answer is not a sale. They also affect human surveys. Asking real people whether they might buy something is not the same as observing what they buy.

Synthetic research adds a second distance: it tries to reproduce the answer to the question, which is itself only one clue about the purchase. A better imitation of a survey response does not remove the need to test the link between that response and the business outcome.

This is the same practical distinction behind testing an AI assistant on the value of the finished work. An impressive intermediate result matters only to the extent that it helps somebody make a better decision or complete a worthwhile task.

For a product team, the relevant success measure might be whether the tool rejects weak concepts without discarding the eventual winner. For a finance team deciding a production run, the required evidence is much stronger. The word “research” should not hide that difference.

The awkward detail is how the measuring stick was chosen

The paper's limitations section says its reference statements were manually optimised for the same 57 surveys. Performance on other surveys remained uncertain, and some demographic patterns were reproduced inconsistently.

This matters because there is more than a language model inside the result. Somebody has chosen how the model's words become numbers. If those choices are improved while looking at the answers they will be judged against, a successful retrospective comparison is not yet a clean test on unfamiliar work.

That does not make the method worthless. It makes the next experiment more interesting.

A team could freeze its prompts, reference statements and scoring method, then test them on concepts kept out of development. The people evaluating those concepts should not see which answer the synthetic panel favoured. Otherwise the process risks quietly steering the human comparison towards the result it hopes to confirm.

The same care is needed with customer groups. Describing a simulated shopper as a particular age or income does not establish that it represents people in that group. If the decision concerns a narrow audience, success on an overall average is insufficient.

The useful role is to improve the question

There is still an attractive business case here. Early product discussions can contain a great deal of untested certainty: customers will love the feature, the premium will be acceptable, the packaging explains itself.

A synthetic panel can help a team turn that certainty into questions. What would make the premium seem worthwhile? Which benefit is unclear? What would a sceptical buyer ask? These reactions are hypotheses to investigate, rather than customer testimony to paste into a board presentation.

Suppose the soap team has six versions of its proposition. It could use the tool to identify repeated confusion, remove jargon and make the options easier to compare. The next human research round would then spend less time discovering that nobody understood the wording.

There is value in that even if the synthetic panel never predicts a single sale. Better questions can make scarce human research time more productive.

The risk is allowing the tool to flatten an unusual idea into a familiar one. A genuinely different product may initially sound odd. If every unconventional response is treated as a defect, the process could reward the concept that is easiest for the model to recognise rather than the one customers will eventually prefer.

Keep the people who can surprise you

The strongest argument for synthetic consumers is not that human research is perfect and expensive. It is that useful research is expensive to repeat, while many early questions are cheap enough to explore in a simulation first.

I would judge a pilot on the decisions it changes. Set aside a small group of new concepts, record the synthetic ranking before seeing the human results, and look closely at disagreements. The expensive error is not merely getting an average score wrong. It is confidently rejecting the product that real customers want, or approving one they will ignore.

Keep an explicit route for evidence the model did not anticipate: observed use, returns, complaints, interviews and actual buying behaviour. A neat set of scores should make those sources easier to prioritise, not easier to dismiss.

The Colgate work gives businesses a serious method to investigate. The sensible ambition is a better-informed product team. The unwise one is a room full of customers who can never walk away.

Comments

No comments yet - start the conversation.

Leave a Comment

Your email address will not be published. No links allowed - keep it kind.