Skip to content

Robots.txt

By Dan Kurtz 1 min read

Robots.txt is a plain-text file at the root of a domain that tells search engine crawlers which URLs they may or may not access. It’s a suggestion (well-behaved bots follow it, malicious ones ignore it), not a security mechanism.

Most sites need three lines: a disallow for admin URLs, an allow for the sitemap, and nothing else. Over-blocking robots.txt is the fastest way to accidentally deindex an entire site.

Related terms

See also on Miss Pepper AI

Where this fits

Part of

Reference Glossary Every term Miss Pepper uses, defined in plain English. Service Comprehensive Guide To Effective SEO Solutions Learn AI-powered strategies, avoid common mistakes, and discover what actually drives organic growth. From technical audits to content…

Keep reading

SEO Alt Text Alt text is a written description of an image, embedded in the HTML img tag. Screen readers read… SEO Schema Markup Schema markup is structured data added to a web page in a machine-readable format (JSON-LD, Microdata, or RDFa)… SEO Crawl Budget Crawl budget is the number of URLs Google will crawl on a given site within a given time…