Robots.txt
Robots.txt is a file that tells search-engine crawlers which parts of a website they are allowed or not allowed to crawl.
What is robots.txt?
Robots.txt is a text file at the root of a website that gives instructions to search-engine crawlers. It can allow or disallow crawling of specific URLs or sections of a site. For B2B marketers, it is mainly relevant when managing crawl budget, migrations, staging environments or large websites.
What does robots.txt do?
A robots.txt file controls crawling, not guaranteed indexing. Blocking a URL in robots.txt prevents compliant crawlers from requesting it, but the URL can still appear in search results if Google discovers it elsewhere. If the objective is preventing a page from being indexed, a robots meta tag is usually the more relevant control.
This distinction matters when diagnosing index bloat. Blocking unwanted URLs from crawling does not automatically remove them from Google's index.
What does a robots.txt file look like?
The file uses directives such as User-agent and Disallow to specify which crawlers the rule applies to and which paths should not be crawled. It can also reference an XML sitemap so crawlers can find important URLs more efficiently.
Why does robots.txt matter for SEO?
For most B2B websites, robots.txt should remain simple. Problems usually appear when important pages, assets or entire site sections are blocked accidentally. That can interfere with crawling and make broader technical SEO work less effective.
Robots.txt is especially important during a website migration. Teams should check that temporary development restrictions are removed before launch and combine that review with correct canonical tags and redirects.
How does robots.txt work in Webflow?
Webflow lets site owners configure robots.txt settings for the website. Before publishing changes, verify that search engines can access the pages intended to generate organic traffic. This should be part of the technical QA for any Webflow website.
Common robots.txt mistakes
Common mistakes include blocking important pages, confusing crawling with indexing, leaving staging restrictions active after launch and using robots.txt as a substitute for proper indexation controls. When a page should consolidate signals with another URL, use the appropriate canonical implementation instead of trying to solve the problem through robots.txt.
Key takeaway
Use robots.txt to manage crawler access, not as a general tool for removing pages from Google. Keep the file simple and check it whenever website structure, migrations or indexation rules change.
