New Study Reveals AI Recommendation Systems Are Highly Vulnerable to Fake Brand Manipulation

Artificial intelligence-powered recommendation systems have grown to become an increasingly influential force shaping how consumers shop online today.

According to data from analytics firm data.ai, large language models (LLMs) already drive roughly 2 percent of all inbound visitor referrals to major retail platforms including Target and Walmart. But a new preprint study released on arXiv by researchers Minghao Luo and Liang Chen exposes a critical flaw in these systems: LLMs can be manipulated with surprising ease. The team’s research tested how simple it is for bad actors to push search-augmented AI recommenders into endorsing completely invented fake brands.

The researchers confirmed that when AI models are fed contaminated, tampered search results, they will routinely turn fabricated products into seemingly credible, authoritative product recommendations.

Luo, a researcher at The Chinese University of Hong Kong, first began investigating this risk after viewing a Chinese television investigative report on the underground industry that produces fake online reviews and manipulated product content. “This is not a hypothetical problem,” he says. The report documented that a completely fake brand could climb to become a top recommendation on mainstream AI systems in just a matter of hours.

To systematically measure this vulnerability, Luo and Chen built a custom testing benchmark called FORGE, an acronym for Fake Online Recommendations in Generative Environments. Rather than attempting to poison live, public search results on the open web, the team recreated the standard end-to-end workflow that most AI recommendation tools rely on to generate answers.

This common process follows a clear pattern: a user requests a product recommendation, the system runs a web search for relevant information, compiles all retrieved pages into an evidence bundle, then feeds that bundle to an LLM to generate a final response. For their experiment, the researchers started with real organic search results and modified them locally, swapping legitimate existing products for entirely invented fake alternatives. They then tested 12 widely used commercial and open-weight LLMs to see if the models would recommend these non-existent brands.

The results were uniformly concerning: every model tested was vulnerable to the manipulation. A single contaminated search page produced a deception rate of up to 27 percent, while replacing the top three retrieved search results pushed the rate of successful fake endorsements as high as 73.8 percent.

Luo shared that he was surprised by how little malicious content was needed to pull off the attack. “You only write one page out of 10,” he explained of the low barrier for bad actors to exploit this flaw.