Browser-based · RFC-aware crawler control
Build robots.txt.Test every crawler rule before you deploy.
Create crawler policies, validate the generated file, test important URLs, inspect AI and search crawler access, and compare your draft against the live robots.txt—without guessing how a rule will behave.
Draft processing stays localContent reviewed Processing boundary
Local rules engine
Robots Control Studio
Saved projects
No named projects saved yet.
No domain has been contacted.
Build, parse, validate and test locally
Build
Policy builder
Starting point
Choose intent, then inspect the rules
Crawler purposes stay separate, especially search discovery, model development and user-triggered retrieval.
Crawler policy
Choose access by purpose; named user-agent groups are generated for you.
Search engines
9 crawlersGoogle, Bing and other traditional search crawlers.
AI search & answer engines
3 crawlersCrawlers used to discover content for answer and search experiences.
AI model development
7 crawlersCrawlers whose documented purpose includes model improvement or training.
SEO & analytics
6 crawlersCommercial crawlers used for audits, backlink indexes and analysis.
Social previews
7 crawlersFetchers that create link cards for social and messaging products.
Archive & research
3 crawlersArchival, monitoring and research-oriented crawlers.
robots.txt expresses crawler preferences. It does not enforce access against crawlers that ignore the protocol.
Path rules
Define the exact site areas this policy should allow or disallow.
SitemapsOptional · 0 configured
URL structure is checked locally. No sitemap is fetched here.
Robots.txt
Valid · 6 lines · 86 B
- 1User-agent: *
- 2Disallow: /admin/
- 3Disallow: /cgi-bin/
- 4Disallow: /*?*sessionid=
- 5Allow: /
- 6
Document status
Valid · 1 group · 4 rules · 86 B · 0 warnings
Ln 1
Build mode. Draft · Valid.
Advanced feature proof
Crawler control you can explain
A verdict is useful only when the tool can show how it reached it.
Every verdict shows its evidence.
The same deterministic matcher powers single tests, batches, the crawler matrix, saved regressions and semantic comparison. If a URL changes state, the winning rule and source line stay visible.
Search discovery, model development and user-triggered retrieval are separate crawler purposes. One choice never silently stands in for the others.
Winning-rule trace
Matched group, source line, match length and competing rules.
Purpose-aware catalogue
Search, AI search, model development and user fetch remain distinct.
Semantic draft diff
Shows behavior changes on tested paths, not just reordered text.
Saved regression tests
Critical URL expectations rerun whenever the rules change.
Live deployment audit
Observed response metadata stays separate from crawler interpretation.
Local deterministic matcher
Normal build, validation, testing and comparison require no API.
How it works
How Robots Control Studio works
Four steps, in the order the studio above supports them.
- 1
Choose crawler policy
Decide which crawler purposes should reach which content.
- 2
Review generated rules
Edit structured groups or the source file without losing custom records.
- 3
Test important URLs
See the matching group, winning rule, line and exact outcome.
- 4
Compare and deploy
Review live impact, download the file and verify production.
Matching
Rules that do not behave the way they read
Each of these is a real file and the verdict the matcher produces for it. Paste any of them into the studio above to check.
Allow one path inside a blocked directory
The longest matching rule wins, so a specific Allow overrides a broader Disallow regardless of order.
Disallow: /private/ Allow: /private/public-info/
✓ ALLOWED⊘ BLOCKED
/private/secret → Blocked · /private/public-info/report.pdf → Allowed (Allow matched 22 octets)
Run in StudioGiving Googlebot its own group
A specific Googlebot group replaces the `*` fallback. If multiple Googlebot groups exist, their equally specific rules combine.
User-agent: * Disallow: /admin/ User-agent: Googlebot Disallow: /search/
Matching the end of a URL
A `$` anchors the match to the end of the path; without it the pattern is a prefix.
Disallow: /*.pdf$
A case that looks blocked but is not
Paths are case-sensitive. This is the rule that most often leaves a directory crawlable by accident.
Disallow: /Admin
Presets
Starting points, not answers
Choose a compact baseline, preview its exact rules in the studio, then adapt it to your site.
General website
4 rules · 1 groupsOpen to search engines, with the usual admin and internal paths kept out.
Preview in StudioSearch open / AI training restricted
8 rules · 8 groupsTraditional search and answer-discovery crawlers stay available; crawler tokens used for model development or broadly reusable training datasets are restricted individually.
Preview in StudioWordPress
5 rules · 1 groupsThe standard WordPress layout, keeping the admin area out while leaving admin-ajax reachable — themes rely on it to render.
Preview in StudioShopify
7 rules · 1 groupsKeeps carts, checkouts and internal search out of the index, which is where duplicate-content problems start on a store.
Preview in StudioNext.js app
4 rules · 1 groupsBlocks API routes and Next internals while explicitly allowing the static assets Google needs to render the page.
Preview in StudioWooCommerce store
9 rules · 1 groupsWordPress plus the store paths: cart, checkout and account stay out, and the add-to-cart links that otherwise generate an endless supply of crawlable URLs are blocked.
Preview in StudioDocumentation site
3 rules · 1 groupsDocumentation earns its traffic by being crawlable, so this blocks almost nothing — only internal search, which produces a result page for every query anyone ever runs.
Preview in StudioSaaS application
6 rules · 1 groupsMarketing pages stay open; the signed-in product does not. Keeps crawlers out of app routes that would only ever return a login redirect, and off the API.
Preview in StudioEcommerce filters
7 rules · 1 groupsFaceted navigation multiplies one category into thousands of near-identical URLs. This blocks the filter and sort parameters that cause it, and deliberately leaves pagination alone.
Preview in StudioStaging site
1 rules · 1 groupsBlocks every crawler. Read the warning: this stops crawling, it does not remove anything already indexed, and it is not access control.
Preview in StudioAllow everything
1 rules · 1 groupsNo restrictions. The shortest valid file that says so explicitly.
Preview in StudioWhich control
robots.txt, noindex, headers and authentication
Almost every robots.txt mistake is really somebody using it for a job one of these other controls does.
| Control | What it stops | What it does not do | Use it when |
|---|---|---|---|
| robots.txt | Crawling — a compliant crawler will not request the URL | Remove pages from search results, hide anything, or stop a crawler that ignores it | You want to keep crawlers out of paths that waste crawl budget or should never be fetched |
| Meta robots noindex | Indexing — the page can be crawled but will not be listed | Prevent the page being fetched, or work at all if robots.txt blocks the crawler from reading the tag | A page must leave search results. Leave it crawlable or the tag is never seen |
| X-Robots-Tag header | Indexing, for any file type — including PDFs and images that cannot carry a meta tag | Prevent fetching, or apply if the response is never requested | You need noindex on non-HTML files, or want to set it at the server rather than per page |
| Password or authentication | Access — nobody without credentials gets the content, crawler or otherwise | Nothing. This is the only entry here that is actually enforcement | The content is genuinely private. Staging sites belong here, not in robots.txt |
| Canonical tag | Nothing. It is a hint about which of several similar URLs is the primary one | Block crawling, block indexing, or bind a search engine to your choice | Duplicate or near-duplicate URLs should consolidate onto one address |
robots.txt
- Stops
- Crawling — a compliant crawler will not request the URL
- Does not
- Remove pages from search results, hide anything, or stop a crawler that ignores it
- Use when
- You want to keep crawlers out of paths that waste crawl budget or should never be fetched
Meta robots noindex
- Stops
- Indexing — the page can be crawled but will not be listed
- Does not
- Prevent the page being fetched, or work at all if robots.txt blocks the crawler from reading the tag
- Use when
- A page must leave search results. Leave it crawlable or the tag is never seen
X-Robots-Tag header
- Stops
- Indexing, for any file type — including PDFs and images that cannot carry a meta tag
- Does not
- Prevent fetching, or apply if the response is never requested
- Use when
- You need noindex on non-HTML files, or want to set it at the server rather than per page
Password or authentication
- Stops
- Access — nobody without credentials gets the content, crawler or otherwise
- Does not
- Nothing. This is the only entry here that is actually enforcement
- Use when
- The content is genuinely private. Staging sites belong here, not in robots.txt
Canonical tag
- Stops
- Nothing. It is a hint about which of several similar URLs is the primary one
- Does not
- Block crawling, block indexing, or bind a search engine to your choice
- Use when
- Duplicate or near-duplicate URLs should consolidate onto one address
robots.txt is not a security control. The file is public, at a predictable address, and lists exactly the paths you would rather nobody visited. Anything genuinely private belongs behind authentication.
Privacy
What runs locally, and what does not
One answer, including the one thing here that genuinely makes a network request.
Runs in your browser
Build · Parse · Validate · Test · Matrix
Rules never leave your browser
Building, validating and URL matching all run on your device. There is no submit step and no request that carries the file you are editing.
Projects stay in this browser
Drafts, saved projects and test cases live in local storage and are never uploaded. Analytics record that a test ran, never the rules, URLs or domains involved.
Requires an explicit network request
Fetch live · Verify deployment
Live fetch sends only the domain
When you ask to read a deployed robots.txt, the address goes to our server, which requests only the root /robots.txt of that public host. Any path you type is discarded first.
The fetch endpoint is fenced in
Private and internal addresses are refused, DNS results are checked before connecting, every redirect is re-validated, and the response is capped and timed out. Nothing fetched is stored.
More detail in how processing works and our privacy policy.
Limits
Standards, limits and compatibility
The values this tool actually enforces, not a rounded-up version.
- Directives matched
- 3
- User-agent, Allow and Disallow. That is the whole of RFC 9309's matching model.
- Crawlers catalogued
- 42
- Across search, AI and research, social previews, SEO tools, advertising and archives.
- Read limit
- 500 KiB
- Google stops reading past this. Rules beyond it are invisible to it, silently.
- Live fetch size cap
- 512 KiB
- Enforced while reading, not trusted from the content-length header.
- Fetch timeout
- 8s
- A site that does not answer in that time is reported rather than left hanging.
- Redirects followed
- 3
- Each hop is re-validated, so a public host cannot redirect us to an internal one.
- Path matching
- Case-sensitive
- Byte for byte. Only the user-agent token is matched case-insensitively.
- Wildcards
- * and $
- Supported by Google, Bing and Yandex. A few older fetchers treat them literally.
Read compatibility detailsIndexing limits, voluntary compliance, wildcard differences and crawler deviations
robots.txt controls crawling, not indexing and not access. A page that is blocked can still appear in search results if other sites link to it — the crawler cannot fetch it to discover a noindex tag, so blocking is the one thing that guarantees the tag is never seen. If a page must leave the index, keep it crawlable and mark it noindex, or put it behind authentication. The file is public at a predictable address, so listing a path here tells anyone who reads it exactly where to look. Compliance is voluntary: well-behaved crawlers honour it and malicious ones ignore it entirely. Google stops reading at 500 KiB, and every rule past that point is invisible to it. Wildcards and $ are extensions rather than part of the original standard, so a few older crawlers treat them as literal characters — Facebook's fetcher is the notable one. Crawl-delay is honoured by Bing and Yandex but not by Google. This studio matches paths exactly as RFC 9309 specifies, but it cannot know how a particular crawler has chosen to deviate, and it does not verify that a crawler claiming to be Googlebot really is one.
FAQ
robots.txt questions
The ones that actually come up.
Robots.txt Generator supporting content: overview, workflow, presets, matching examples, control comparison, privacy, limits, frequently asked questions and related tools.