How do search engines work?

The three stages in plain English

Google's own documentation splits Search into three stages. Crawling is downloading pages with automated programs called crawlers. Indexing is analysing what was downloaded and storing it in a huge database called the index. Serving is choosing which indexed pages to show for a particular search.

The stages run in order, and a page can stop at any of them. A page that is never crawled cannot be indexed. A page that is not indexed cannot be shown. Most "why am I not ranking?" problems are really "why did this stage not happen?" problems.

This is the mental model behind every SEO decision. Our technical SEO audit is a structured way of checking each stage on your own site.

Stage one: crawling

Google's crawler is called Googlebot. Google says it finds URLs through links on pages it already knows, through pages it visited before, and through sitemaps you submit. Googlebot decides which sites to crawl, how often and how many pages to fetch, and it slows down if your server struggles.

Some discovered URLs are never crawled. Pages blocked by robots.txt rules are skipped, and so are pages behind a login. Server errors and network problems can also stop a visit. During the crawl, Google renders the page with a recent version of Chrome, because some content only appears after JavaScript runs.

The practical lesson: give every important page at least one normal link from another page. Details are in crawlability and indexation.

Stage two: indexing

After crawling, Google analyses the page: its text, images, headings, language and what it appears to be about. It stores this in the index. Google does not index everything it crawls. Its documentation names low-quality content, rules that block indexing and designs that make indexing difficult as common causes.

Google also looks for duplicates. It groups similar pages into a cluster and selects one as the canonical page, the version it will show. If you do not say which version you prefer, Google chooses for you, and it may not choose the one you wanted. The canonical URL is how you state your preference.

Technical requirements for indexing are short. Googlebot must not be blocked, the page must return a working response, and it must contain indexable content. Meeting them makes a page eligible, not guaranteed.

Stage three: serving and ranking

When someone searches, Google looks in its index for pages that match and orders them. Its documentation says relevance depends on hundreds of factors, including the searcher's location, language and device. It does not publish the factors or their weights.

What Google does publish is what it is trying to reward: helpful, reliable content made for people. It also says good page experience, including the Core Web Vitals, is used by its ranking systems. But it states it will show the most relevant content even when page experience is weaker.

The result also depends on the searcher's purpose. A query like "solicitor fees" and one like "best solicitor in Leeds" call for different pages. Matching search intent is part of how relevance is judged. See keyword research.

Where AI answers fit in

AI assistants add a layer on top. Many of them retrieve pages from a search index first, then write an answer from what they found. That means a page must still be crawlable and indexable before an assistant can quote it.

Being named in the answer is a different goal from ranking in a list, which is why we treat it as its own discipline. AEO vs SEO vs GEO explains the difference, and our AEO service covers the work.

How to find which stage is failing

Use Google Search Console. Its URL Inspection tool shows whether Google can crawl a specific page, which version it rendered and which canonical it selected. The Page indexing report lists URLs that were not indexed and the reason.

Read the statuses literally. "Discovered, currently not indexed" means Google knows the URL but has not crawled it yet. "Crawled, currently not indexed" means Google visited but did not add it. The first is a crawl-capacity or priority problem. The second is usually a content or quality problem.

Start with a few important pages and trace them through the three stages. One worked trace teaches more than a site-wide report you cannot yet read.

What changes, and what does not

Google updates its ranking systems often, and it adds features such as AI summaries. The three stages have stayed the basic model. Guidance pages we cite here were last updated between 2024 and 2026, which is a reminder to check the source rather than trust old advice.

What does not change is the advice. Make pages reachable, useful and trustworthy, and check that Google sees what a user sees. Be wary of anyone who claims to know the exact ranking recipe.