Glossary / SEO, GEO & AEO

robots.txt

robots.txt is a plain-text file at your site's root that tells crawlers which paths they may and may not fetch. It's a politeness convention, not security — well-behaved bots honor it, attackers ignore it, so it must never be the thing hiding sensitive URLs. Used well, it keeps crawl budget focused on the pages that deserve indexing.