robots.txt
robots.txt is a text file at the root of a website that tells crawlers which pages they may or may not fetch, by user agent, under the Robots Exclusion Protocol.
Updated · Citable
What robots.txt means
Each group starts with a User-agent line naming a crawler (or * for all) followed by Allow and Disallow rules. Well-behaved crawlers, including Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, follow it. It controls crawling, not indexing: a blocked page can still be indexed from links, and noindex is the tool for keeping a page out of results.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml