On this page
If you've tried an AI visibility tool, you may have seen your score jump from one week to the next. Your store didn't change. Your product pages didn't change. But the number did.
That isn't always the tool's fault. AI answers really do change from one ask to the next. Once you know how much, you can tell a real change from noise, and measure the things that matter for a store.
AI answers change almost every time you ask
SparkToro had volunteers ask ChatGPT, Claude and Google's AI the same questions for brand recommendations, nearly 3,000 times in total. It found less than a 1 in 100 chance that ChatGPT or Google's AI would give the same list of brands twice. The chance of the same list in the same order was lower still.
The websites AI assistants name as sources change too. Profound, a company that sells AI tracking, compared answers to about 80,000 questions per assistant, one month apart. Depending on the assistant, 40% to 59% of the websites named as sources in the second month hadn't been named in the first.
So if a tool asks each question once and reports where you came up, much of what you're seeing is chance.
Why a single ranking misleads
Many tools report a position. For example: you're #3 for "best moisturizer for sensitive skin". In a list of links on Google, a position means something, because the list is fairly stable from day to day. In an AI answer, it doesn't. The same question asked again can put you first, fifth or nowhere.
SparkToro's author put it bluntly: any tool that claims to give a ranking position in AI answers is "full of baloney."
What holds steady: how often you show up
The same study found something more useful. Some brands showed up again and again, even when the order kept changing. Across 994 answers to questions about headphones, written by 142 different people, Bose, Sony, Sennheiser and Apple each appeared in 55% to 77% of them.
That's the number worth tracking: how often your store appears, out of many answers. SparkToro concluded that a visibility percentage, measured across many questions each asked many times, is a reasonable thing to measure. If your store shows up in 2 out of 10 answers this month and 5 out of 10 next month, that's a change you can trust far more than a jump from #4 to #2.
What to measure instead
How often you're named, on your buyers' questions. Use the questions your own customers ask, in their words. Skip generic ones like "best skincare brand". Ask each question several times, on more than one assistant. Count the share of answers that name your store.
Who gets named when you don't. For each question, note which stores show up in your place. That tells you who you're really competing with in AI answers. It's often a different list from your Google competitors.
Whether the facts are right. Ask about your store by name. Check what the answers say about your prices, shipping and return policy. An answer that names you with the wrong price can do more harm than good. We covered one study on this in AI got basic facts wrong for 64% of retailers.
Visits from AI assistants. In Shopify, open Analytics, then Reports, and look at the report of sessions by referrer. Shopify calls each visit a session. A referrer is the website a visitor came from. Visits from AI assistants show up there under addresses like chatgpt.com and perplexity.ai. Not every visit from an AI assistant is labeled this way, so treat the number as a floor.
Your own starting point. Write all of this down before you change anything. Then measure again after each change, with the same questions. Comparing your store with itself over time tells you more than any score compared with other stores.
How to read a change
A few rules of thumb help:
- One answer is not a result. A single ask can't tell you whether anything changed.
- Look at the share, not the position. Going from 2 in 10 answers to 5 in 10 is a real signal. Moving from #4 to #2 once isn't.
- Expect the sources to rotate. A website dropping out of the answers for a month doesn't always mean it's gone for good.
- Change one thing at a time when you can. Otherwise you won't know what made the difference.
To check why other stores keep getting named instead of yours, see why ChatGPT recommends your competitors, and not you. And our AI visibility audit guide walks through setting up your starting point.
To see what an AI assistant can read on your product pages, apply for a free check.
What this doesn't settle
Both studies come from people connected to AI tracking companies. SparkToro's author is connected to one and says the study wasn't reviewed by outside researchers. Profound sells tracking and hasn't published its raw data. The studies looked at brands and sources in general, not at Shopify stores. And no measurement removes the chance in AI answers. It only shows you how big the chance is, so you can see past it.
