How search works now: crawl, index, rank, and an AI layer on top

Organic search is still the biggest source of traffic for most websites. BrightEdge's 2019 study across thousands of sites put it at 53.3% of trackable traffic, more than paid search and social media combined.

It's also very uneven. When Ahrefs checked the roughly 14 billion pages in its index in 2023, 96.55% got no traffic from Google at all.

Being online isn't the same as being found. The gap comes from how search engines work, and AI answers now add one more step on top.

Stage 1: crawl

A search engine sends a crawler to fetch pages and follow the links on them. A page that isn't crawled never gets to the later stages.

Crawlers don't fetch everything they could. Google sets a crawl capacity limit for each site: "If the site responds consistently and its response times … remain stable or improve, the limit goes up, meaning more connections can be used to crawl." When a site slows down or returns errors, Google crawls less. Google writes this guidance for large sites, but the idea applies to a store with thousands of product and variant pages: a fast, stable site gets more of its catalog fetched, more often.

Every AI assistant that searches the web runs its own crawler too, and each one decides for itself what to fetch.

Stage 2: index

Fetched pages are processed and stored: the text, the images, the structured data, what the page is about. Only indexed pages can show up in results, including Google's AI features. Google says a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear as a link in AI Overviews or AI Mode.

Not everything fetched gets indexed. Duplicates, thin pages and pages marked not to be indexed are left out.

Stage 3: rank

For each search, the engine orders the indexed pages it judges most relevant and useful. There's no fixed checklist. Google adjusts its ranking systems constantly and announces the bigger changes. In 2025 it announced four: core updates in March, June and December, and a spam update in August. Each one moved some sites up and others down without them changing anything.

Stage 4: the answer

AI search adds a step. Instead of handing over a ranked list, the engine, or an assistant like ChatGPT or Perplexity, reads a set of pages and writes an answer, citing some of them.

Ranking still feeds this step, but less directly than you might expect. Writesonic analyzed more than a million Google AI Overviews in 2025 and found that 40.58% of citations came from pages in the top 10 results. Ahrefs, looking at 4 million AI Overview URLs in early 2026, found that 38% of cited pages ranked in the top 10 for the same search, down from 76% in its mid-2025 analysis. Part of that drop reflects how Ahrefs now measures. Part reflects how AI Overviews work: they split a question into several related searches and draw from all of them.

So ranking helps, but most citations come from pages outside the top 10 for the question asked. What gets a page cited is answering part of the question clearly.

What this means for a store

Each stage fails in its own way for a product page.

• Crawl: a slow or unstable site gets less of its catalog fetched, and less often, so prices and stock in an answer can lag behind your site.
• Index: duplicate variant pages and thin product pages are left out before they're ever considered.
• Rank: one algorithm update can change which product pages show up, without any change on your end.
• Answer: the assistant cites the page that best answers the shopper's question: what the product is for, who it fits, what it's made of. This is the stage where your product data does the most work.

The first three stages need to stay healthy. The fourth is where a store has the most to gain, because it rewards what only the store knows about its own products.

What you can't control

When Google ships an update, how each assistant chooses its sources, and the fact that the same question can be answered differently twice. These studies are averages across millions of searches. Your category will differ.

A quick check

1. In Google Search Console, open the Pages report and see how many product pages aren't indexed, and why.
2. Open Crawl stats and check whether your average response time is steady or rising.
3. Ask ChatGPT, Perplexity and Google AI Mode a question your best product should answer. Note which pages get cited, and whether any of them rank in Google's top 10 for the same search.