What is crawlability (and indexation)?

Two different things: crawling and indexing

Crawling is a search engine's crawler downloading your page. Indexing is the search engine deciding to analyse that page and store it so it can be shown. People mix the two up, and the difference decides which fix you need.

If a page is not crawled, the problem is access: a block, a missing link, a broken response. If it is crawled but not indexed, the problem is usually the page itself: duplicate, thin or low-quality content, or an instruction telling Google to leave it out.

Google's Page indexing report labels these directly. "Discovered, currently not indexed" means the URL is known but not yet crawled. "Crawled, currently not indexed" means Google visited and did not add it. For the wider process, see how search engines work.

The controls that affect crawling

The robots.txt file sits at the root of your domain and tells crawlers which URLs they may request. Google says it is used mainly to avoid overloading your site. Crawlers that choose not to follow it can ignore it, so it is not a security tool.

Internal links are the other main control. Google says it finds most pages by following links, so every important page needs at least one link from another page. Google also says it parses only anchor elements with an href. A button or menu that relies on a script click may never be followed.

A XML sitemap lists the URLs you want crawled. Google says smaller sites of roughly 500 pages or fewer with thorough internal linking may not need one, and that submitting one does not guarantee crawling. Keep it limited to indexable, canonical pages.

The controls that affect indexing

A noindex directive, set as a robots meta tag or an X-Robots-Tag HTTP header, tells Google to drop the page from results. The header works for files such as PDFs. Google does not support noindex inside robots.txt.

The most common conflict: blocking a page in robots.txt and expecting noindex to work. If Googlebot cannot crawl the page, it never sees the tag, and the URL can still appear if other pages link to it. To remove a page from search, allow crawling and use noindex.

The canonical URL tells Google which version of duplicate pages to keep. Google says redirects and rel="canonical" are strong signals, and listing a URL in a sitemap is a weak one. It recommends a self-referencing canonical on the canonical page and absolute URLs.

JavaScript and rendering

Google renders pages using a headless version of Chrome before indexing, but only for pages it is allowed to crawl and that return a normal response. If robots.txt blocks a script file, Google will not render the content that depends on it.

Single-page apps bring specific traps. Fragment-based routing such as "#/products" is not crawlable, so use real URLs. An error view that still returns a success code becomes a soft 404. Google also notes that it may skip rendering pages carrying noindex, so a script that removes the tag may never run.

Server-side rendering is still recommended by Google because it helps users and crawlers. If your web development setup depends on client-side rendering, test with URL Inspection.

Crawl budget: who needs to care

crawl budget is the set of URLs Google can and wants to crawl on your site. Google says its guidance is for sites with roughly a million pages that change weekly, sites with about ten thousand pages that change daily, or sites with many "Discovered, currently not indexed" URLs.

If you run a service business with a hundred pages, crawl budget is not your problem. Fix links, duplicates and quality instead. For large shops, Google recommends consolidating duplicates, blocking unimportant URLs in robots.txt, returning 404 or 410 for removed pages and keeping redirect chains short.

How to diagnose and fix it

• Open the Page indexing report in Google Search Console and group URLs by reason.
• Inspect a representative URL for each reason with URL Inspection and read what Google saw.
• For blocked pages, fix robots.txt. For orphan pages, add links. For duplicates, choose a canonical.
• For "Crawled, currently not indexed", improve the page: merge thin pages, add real expertise, or remove the page.
• Resubmit the sitemap and request indexing for key pages, then recheck in a few weeks.

Measuring and what is changing

Track the count of indexed pages against the pages you want indexed. Ignore the total if it includes pages you never wanted in search. A healthy site has a high share of its valuable pages indexed and a clear reason for each excluded one.

AI assistants add pressure here. The crawlers they use are separate, so check what you allow them to access. Our guide to llms.txt and the AI crawler entry cover this, and AEO vs SEO vs GEO places it in context.