---
title: "robots.txt"
description: "robots.txt controls crawler access to a site. Learn the Disallow, Allow and Sitemap directives and the difference between blocking and noindex."
url: "https://nodea.io/en/glossary/robots-txt"
lang: "en"
updated: "2026-07-13"
---

# robots.txt

> robots.txt is a text file placed in a domain's root directory that tells search engine crawlers which parts of the site they may or should not visit.

**robots.txt** is a text file placed in a domain's root directory (at *example.com/robots.txt*) that instructs search engine [crawlers](https://nodea.io/en/glossary/crawler) which parts of a site they may visit and which they should skip. It is part of the Robots Exclusion Protocol — a voluntary standard that reputable bots such as [Googlebot](https://nodea.io/en/glossary/googlebot) respect. It is usually the first file a crawler requests before it begins scanning a site.

## How robots.txt works

The file consists of rules assigned to specific crawlers. The key directives are:

- **User-agent** — names the crawler the following rules apply to (an asterisk means all),
- **Disallow** — a path the crawler should not fetch,
- **Allow** — an exception permitting access to a resource inside a blocked directory,
- **Sitemap** — the address of the XML [sitemap](https://nodea.io/en/glossary/xml-sitemap).

For example, the rule *Disallow: /admin/* tells crawlers not to scan the admin panel. The crucial point is that robots.txt controls **crawling**, not [indexing](https://nodea.io/en/glossary/page-indexing) — the two are different things.

## robots.txt in practice

robots.txt is used mainly to conserve crawl budget: you block resources with no value to search engines, such as internal site searches that spawn endless URLs, login panels or technical files. A common and dangerous mistake is trying to hide a page from Google with *Disallow* alone — a blocked URL can still surface in results if links point to it. To exclude a page from the index, use the [robots meta tag](https://nodea.io/en/glossary/meta-robots) with a *noindex* value, not robots.txt. When launching a new site, always check that the file is not accidentally blocking everything with *Disallow: /*, a classic reason a site vanishes from Google.

### Does blocking a page in robots.txt remove it from Google?

No. robots.txt discourages a crawler from fetching a page, but a blocked URL can still appear in results if other pages link to it. To remove a page from the index, use a noindex meta tag or X-Robots-Tag header and do not block it in robots.txt first.

### Where must the robots.txt file live?

Always in the domain's root, at example.com/robots.txt. Crawlers look for it only there — a file placed in a subdirectory is ignored. Each subdomain needs its own robots.txt.

## Related

- [Crawler](https://nodea.io/en/glossary/crawler)
- [Googlebot](https://nodea.io/en/glossary/googlebot)
- [Page indexing](https://nodea.io/en/glossary/page-indexing)
- [XML Sitemap](https://nodea.io/en/glossary/xml-sitemap)
- [Meta robots](https://nodea.io/en/glossary/meta-robots)

---

Languages: [pl](https://nodea.pl/pl/slownik-pojec/robots-txt.md)  
Site index: [llms.txt](https://nodea.io/en/llms.txt)