Robots.txt
A file that tells search engine crawlers which pages they can and cannot access.
Why it matters
Robots.txt controls crawl access and helps manage crawl budget.
Want to understand how this applies to your website?
Get an SEO assessment tailored to your specific situation.