Robots.txt declares paths a crawler should not visit. It can manage crawling of public content, but it is not a security control for sensitive information.
The demo rule switch blocks the robot's route to /private. After editing rules, verify the served file and the target crawler's interpretation. If removal from the index is the goal, consider a separate noindex directive.
When to use
Use it to design and test exclusions when crawlers repeatedly fetch low-value paths.