Crawler
A crawler is a program that requests URLs over HTTP, stores the responses, extracts the links inside them, and adds those links to a queue of URLs to request next. Search engines use crawlers to discover and fetch the pages they later index.
Crawler, spider, robot and bot all mean the same thing. Googlebot, Bingbot and Amazonbot are individual crawlers, each identified by a user-agent string in the requests it sends.
The loop is simple and never changes: take a URL from the queue, check whether
robots.txt permits fetching it, send the request, record the status code and
body, extract the links, enqueue the new ones. Everything a search engine knows
about the web arrived this way.
Two details matter in practice. A crawler limits how fast it requests pages from one host, because a crawler without politeness limits is indistinguishable from an attack. And any client can claim to be Googlebot in its user-agent string, so a reverse DNS lookup on the requesting IP is the documented way to verify a genuine one.
Related: crawl budget, and the full crawling guide.