robots.txt
robots.txt is a plain text file at the root of a host that tells crawlers which paths they may request. Standardised as RFC 9309, it controls fetching only. It does not remove pages from an index, and compliance by any given crawler is voluntary.
The file lives at exactly one place per host — https://example.com/robots.txt
— and applies to that host and protocol only. It contains groups of rules, each
naming a user-agent and listing Allow and Disallow paths.
The most common and most costly misunderstanding: robots.txt stops a fetch, it
does not stop a listing. If other pages link to a URL you have disallowed, a
search engine still knows the URL exists and can show it in results without a
snippet, because it was never permitted to look inside.
The corollary is that combining a Disallow rule with a noindex tag on the
same URL defeats the noindex entirely. The crawler never fetches the page, so
it never reads the directive. To remove a page from an index, allow the fetch
and serve noindex.
Compliance is a convention rather than an enforcement mechanism. Well-behaved crawlers honour the file; anything that intends to ignore it will. Access control that actually matters belongs behind authentication.