What robots.txt is for
Robots.txt manages crawler traffic so your site is not overloaded. Google says it is not a way to keep a web page out of Google.
URL blocked by robots.txt
A rule in your robots.txt file stops Google crawling this URL. Remove the rule if the page should be in search. If you want it kept out of Google, robots.txt is the wrong tool; use noindex.
Diagnose it
Symptoms, checks and fixes
Common symptoms
- URLs listed under "Blocked by robots.txt" in the Page indexing report.
- A whole section, or the whole site, stopped appearing after a robots.txt change.
Checks to run
| Check | How to tell | Fix |
|---|---|---|
| Which rule blocks the URL? | Open yoursite.com/robots.txt and find the Disallow line that matches the path. | Delete or narrow the rule so it no longer matches pages you want crawled.[1][2] |
| Is a staging rule live? | Look for "Disallow: /" under "User-agent: *", often left over from a development site. | Replace it with the live site rules before or straight after launch.[2] |
| Do you actually want it hidden? | Decide whether the page should appear in Google at all. | To hide it, allow crawling and add a noindex rule, because Google must crawl the page to see noindex.[3][1] |
How we work together
Two ways to get started
Website Subscription
Your website, fully managed — no big upfront cost.
We design, build, host, maintain and keep improving your site for one simple monthly fee.
- Custom-designed & built for your brand
- Hosting, security & maintenance included
- Ongoing edits & content updates
- Continuous SEO & performance improvements
- Scale up or down as you grow
Fixed Project
A one-off custom build you own outright.
A defined scope and a single fixed quote — delivered once, and yours to keep.
- Bespoke design & development
- Defined scope & timeline
- You own the website & code
- Optional care plan after launch
- Custom-quoted to your needs
Sources
- [1] Page indexing report (Search Console Help). Checked 9 October 2026.
- [2] Google Search Central: Introduction to robots.txt. Checked 10 October 2026.
- [3] Google Search Central: Block search indexing with noindex. Checked 10 October 2026.
Ready to get started?
Get a free, no-obligation quote for your project. We respond within 24 hours.