Robots.txt tells crawlers which URLs they may fetch. Noindex tells Google not to keep a page in search results. They solve different problems, and mixing them up is one of the most common ways to keep a page in Google by accident.
Crawling means a bot fetches a URL. Indexing means Google stores the page so it can appear in results. Robots.txt works on the first step. Noindex works on the second.
Google describes robots.txt as a way to tell compliant crawlers which URLs they may access, mainly to manage crawl traffic. It says plainly that robots.txt is not a mechanism for keeping a web page out of Google.
If a URL is disallowed in robots.txt but other pages link to it, Google may still index the URL. Google's robots.txt specification says it can show the URL in results without a snippet. The page content is not read, so the result has no description.
This surprises many teams. They block a section, then find its URLs listed in search. The block stopped the fetch. It did not remove the URL from Google's knowledge.
Noindex is a rule on the page itself. You can add it as a meta tag in the head of the HTML, or send it as an X-Robots-Tag HTTP header. The header also works for files such as PDFs, images and video. Google says that once Googlebot sees the rule, it drops the page from results, even if other sites link to it.
Googlebot has to fetch the page to see that rule. If robots.txt blocks the URL, the crawler never sees the noindex tag or header. The page can then stay in results when other pages link to it. Google also says it does not support noindex inside robots.txt.
The file must be named robots.txt and sit at the root of the host it covers. A site can have only one, and it applies only to the protocol, host and port where it is posted. It must be UTF-8 plain text.
Google enforces a limit of 500 KiB. Content beyond that is ignored. Google generally caches the file for up to 24 hours. If the file returns a 4xx error, Google treats it as if no valid file exists, with 429 handled differently. If it returns a 5xx error, Google stops crawling the site for the first 12 hours, then uses the last cached version for up to 30 days.
Pick the tool by the goal. If you want a URL out of search results, allow crawling and add noindex, or password protect it. If you want to reduce crawling of sections that should never be fetched, use robots.txt.
To check a page, use the URL Inspection tool in Search Console to see the HTML Googlebot received and confirm the noindex rule is there. The robots.txt report in Search Console helps you check the file. The Page Indexing report shows pages where Googlebot found a noindex rule. For urgent removals, Google points to its removals documentation instead of waiting for a recrawl.
Sources were last checked on October 13, 2026. Documentation changes, so confirm any detail that matters to a decision against the source itself.
Indexive Research. (2026, July 8). What robots.txt and noindex each really control. Indexive AI. https://indexive.ai/research/robots-txt-vs-noindex
Straight answers about how we work. Still have a question? The free audit is the fastest way to get one answered for your site.
No. Google says robots.txt is not a mechanism for keeping a page out of Google. A disallowed URL can still be indexed if other pages link to it, usually without a snippet.
A free execution audit: your prioritized backlog, effort estimates, and what we'd ship first. Read-only access, no contract.