Three AI Models Picked Different Products and Revealed a New Frontier in Agentic Commerce

The rise of agentic commerce has introduced a complex, opaque, and highly influential layer between the modern shopper and the retail store: the large language model (LLM). As Shopify recently enabled a direct, keyless connection for AI agents to its vast product catalog, the mechanics of how these agents make purchasing decisions have come under intense scrutiny. A controlled experiment involving nearly 2,000 simulated shopping decisions across three Claude models—Haiku 4.5, Sonnet 5, and Opus 5—reveals that when human bias is removed, AI models rely on their own internal logic to favor certain products, often ignoring catalog rankings in favor of their own programmed "safe" preferences.

The New Architecture of Shopping

On October 13, 2026, Shopify fundamentally altered the landscape of e-commerce by announcing Shop MCP (Model Context Protocol). This integration allows AI agents and third-party applications to query the Shopify catalog, assemble shopping carts, and facilitate checkout via Shop Pay without the traditional friction of API keys. By providing a streamlined, low-latency bridge to over a billion products, Shopify has effectively invited AI to act as a primary procurement layer for consumers.

However, the transition from a search result to a completed transaction requires a decision-making agent. While retailers have spent decades optimizing for human search engine behavior—focusing on SEO, keyword density, and site architecture—the emergence of agentic commerce shifts the goalpost. The agent does not simply display a list; it parses the list and makes a selection. Because these agents operate on language models, the final choice is governed by the underlying weights and biases of the model itself.

Experimental Methodology and Data

To understand the nature of these hidden decision-making processes, a systematic test was conducted using a standardized set of 20 products for six distinct categories: coffee beans, water bottles, backpacks, running shoes, wireless earbuds, and yoga mats. Each search was performed as an anonymous user, and the resulting top 20 list was fed to three Claude models—Haiku 4.5, Sonnet 5, and Opus 5—50 times each per category.

The models were given a stark, minimalist instruction set: "You are a shopping agent. A person has asked you to buy something for them. Choose the one product you would buy for them." No shopper history, budgetary constraints (in the initial tests), or user preferences were provided. This "black box" approach was designed to isolate the inherent decision-making tendencies of the models.

The results demonstrated a lack of consensus that challenges the assumption that AI agents will behave predictably. Across the six categories, the three models failed to agree on a single top product. Furthermore, while the models were remarkably consistent with their own internal logic—Haiku and Sonnet frequently gravitated toward products with high review counts—they were entirely inconsistent with one another. When the underlying model was swapped, the preferred product changed, and in many instances, the store receiving the potential order shifted as well.

The "Safety" Bias and the Role of Heuristics

One of the most significant findings involves the behavior of the Claude Opus 5 model, which frequently justified its selections by labeling them "safe" or "the safest crowd-pleaser." In 222 out of 300 instances, Opus prioritized products that represented a perceived default, such as a medium roast coffee over a dark roast, regardless of review counts.

This behavior suggests that LLMs are not merely processing data; they are applying probabilistic heuristics to minimize the perceived risk of a "bad" recommendation. By contrast, Haiku and Sonnet leaned heavily on quantitative data points like review counts, often citing figures that were not present on the storefronts themselves. In the case of a specific stainless steel water bottle, two models cited an astronomical figure of 447,221 reviews, a number far exceeding the actual count on the manufacturer’s site. This highlights a critical vulnerability: agents are susceptible to hallucinated or misparsed data points within the catalog feed, and they lack the external verification tools to correct these inaccuracies.

The Impact of List Order and Contextual Constraints

The experiment also tested the influence of list positioning. When the top-ranked product in a catalog search was moved to the bottom of the list of 20, the likelihood of it being selected dropped significantly across all three models. Despite the models citing features like price or ingredients in their justifications, not a single one mentioned the product’s position in the list. This suggests a "positional bias" inherent to how models process sequential data, a factor that store owners currently have no way to audit or counter.

However, the influence of the model is mitigated when specific constraints are introduced. When the search request was modified to include a budget ("under $25") and a specific need ("oily skin"), the models abandoned their preference for the top-rated, higher-priced items and shifted to lower-cost alternatives. This indicates that while agents possess an internal "default" logic, they are highly responsive to explicit, structured constraints. The selection process became a matter of keyword matching—for instance, Opus favored a sunscreen that specifically mentioned "without clogging pores," regardless of its lower star rating compared to other budget-friendly options.

Implications for E-commerce and Retailers

The shift toward agentic commerce creates a new reality for digital storefronts: they are no longer just competing for the eyes of consumers, but for the "preference" of algorithms. Traditional SEO remains relevant, but its impact is mediated through a layer of AI reasoning that is entirely invisible to the retailer.

Several key implications arise from these findings:

1. The Obsolescence of Traditional Ranking Audits: Stores can no longer rely on standard ranking reports. Because every agent may be running on a different model—or a different version of a model—there is no longer a single "first page" of results.
2. The "Reasoning" Gap: The justifications provided by agents are often based on fragmented, catalog-level data. Retailers must ensure that their product feeds are not just SEO-optimized, but "reasoning-optimized"—providing clean, structured, and accurate descriptive text that an agent can parse without hallucinating.
3. The Loss of Brand Narrative: In a race to be the "safest" or most "statistically likely" choice, the unique value propositions of niche brands may be flattened. If an agent is programmed to prioritize safety, it may systematically filter out products that are bold, innovative, or specialized.

Conclusion: A New Era of Algorithmic Competition

The ability for any company to build a shopping agent using Shopify’s open MCP connection democratizes access to commerce, but it also decentralizes the power of the storefront. The decision-making layer between the user and the store has become a black box. Retailers must now adapt to a landscape where their success depends on how well their product data aligns with the generalized, often conservative, internal logic of AI models.

As the industry matures, there will likely be calls for more transparency in how agents reach their conclusions. For now, the burden remains on the retailer to curate high-fidelity product information that can withstand the scrutiny of a cold, analytical, and highly unpredictable AI agent. The competitive advantage will go to those who can master the art of being the "obvious" choice, not just to the human eye, but to the latent space of the model.