robots.txt is a plain-text file at a site’s root that publishes crawler instructions, such as which paths a user-agent group may or may not fetch. A crawler typically checks a URL like https://example.com/robots.txt before requesting pages on that host.
Reading the file correctly
Rules apply to user-agent groups and URL paths according to the Robots Exclusion Protocol. A Disallow line is not a password-protected boundary; the URL may still be accessible to a browser or another client. Conversely, a path omitted from the file is not automatically authorized for every use. Review the site’s terms and access policy as well.
For site owners, keep the file syntactically valid and test how intended crawlers interpret the rules. Do not list private URLs as a way to hide them; use authentication for restricted content.
Directives such as Crawl-delay are not standardized by RFC 9309, so crawler support differs. Consult the documentation for the crawler you use.
Leave a Reply