Robots.txt

robots.txt

A site file that tells crawlers which paths they should not request.

···

Robots.txt declares paths a crawler should not visit. It can manage crawling of public content, but it is not a security control for sensitive information.

The demo rule switch blocks the robot's route to /private. After editing rules, verify the served file and the target crawler's interpretation. If removal from the index is the goal, consider a separate noindex directive.

When to use

Use it to design and test exclusions when crawlers repeatedly fetch low-value paths.

Open as page ↗

Suggest an entry

Tell us about a term, effect or component that’s missing. We read every suggestion.