Robots.txt

robots.txt

A site file that tells crawlers which paths they should not request.

···
html
<div class="robot"><div class="rule"><b>robots.txt</b><code>User-agent: *<br><span id="directive">Disallow: /private</span></code><button id="toggle">TOGGLE RULE</button></div><div class="route"><div>● crawler</div><div class="arrow">→</div><div class="target" id="target">/private <strong>BLOCKED</strong></div></div></div>
css
.robot{width:min(90%,590px);display:grid;gap:clamp(8px,2vmin,18px);font:800 clamp(10px,max(1.8vmin,1vw),15px) sans-serif;color:var(--fg)}.rule{display:grid;gap:5px;border:1px solid var(--line);padding:clamp(9px,2vmin,18px);background:var(--surface)}.rule b{color:var(--accent)}code{font:700 clamp(10px,max(1.7vmin,1vw),15px)/1.35 monospace}.rule button{justify-self:start;background:var(--fg);color:var(--bg);border:0;padding:5px 10px;font:inherit;cursor:pointer}.route{display:flex;justify-content:space-between;align-items:center}.arrow{font-size:clamp(20px,4vmin,35px);color:var(--accent);animation:walk 1.8s ease-in-out infinite alternate}.target{border:2px solid var(--accent);padding:clamp(6px,1.5vmin,13px);display:grid;gap:3px}.target strong{color:var(--accent);font-size:max(10px,1vw,.75em)}.target.open{border-color:var(--accent-2)}.target.open strong{color:var(--accent-2)}@keyframes walk{to{transform:translateX(14px);opacity:.35}}@media(min-width:700px){body > :not(script){zoom:1.55}.map,.feed,.linkmap{height:48%}}
js
const target=document.querySelector('#target'),directive=document.querySelector('#directive'),button=document.querySelector('#toggle');let blocked=true;function show(){target.classList.toggle('open',!blocked);target.querySelector('strong').textContent=blocked?'BLOCKED':'OPEN';directive.textContent=blocked?'Disallow: /private':'Disallow: /tmp'}button.addEventListener('click',()=>{clearInterval(timer);blocked=!blocked;show()});const timer=setInterval(()=>{blocked=!blocked;show()},2500);

Robots.txt declares paths a crawler should not visit. It can manage crawling of public content, but it is not a security control for sensitive information.

The demo rule switch blocks the robot's route to /private. After editing rules, verify the served file and the target crawler's interpretation. If removal from the index is the goal, consider a separate noindex directive.

When to use

Use it to design and test exclusions when crawlers repeatedly fetch low-value paths.

Open as page ↗