Webflow Robots.txt

A Webflow robots.txt file is a public text file at the root of a Webflow site that tells compliant crawlers which URLs they may or may not access. It controls crawling behavior, not whether a page is guaranteed to appear in search results.

How does robots.txt work?

Crawlers request the file at /robots.txt before accessing site URLs. Rules are grouped by User-agent and usually use Disallow or Allow directives for paths. This makes robots.txt a traffic-control layer for website crawling, but it relies on bots choosing to follow the rules.

Is robots.txt the same as noindex?

No. A blocked URL cannot be crawled, but it may still be discovered through links and appear in search results without a useful snippet. If a page should not be indexed, use an appropriate noindex control while allowing the crawler to access that instruction. Treat crawl access and search indexing as separate decisions. Never use robots.txt to protect private information.

How do you configure robots.txt in Webflow?

In Webflow, add robots.txt rules under Site settings, SEO, and Indexing, then save and publish the site. The file applies at the domain level, so test it on the final production hostname. A careful Webflow SEO implementation uses specific rules and avoids broad blocks unless the whole site or directory truly should be inaccessible to crawlers.

How does robots.txt affect AI Search?

AI search platforms use different crawlers for search retrieval, training, or other purposes, and bot identities and policies can change. Blocking a relevant crawler can reduce its ability to retrieve and cite site content. An AI Search strategy should document which bots support each business goal, review current first-party crawler documentation, and avoid treating one global rule as sufficient governance.

How should robots.txt be tested?

Fetch the live file after every change, confirm its HTTP status, validate the syntax, and test representative allowed and blocked URLs. Then monitor crawl behavior rather than assuming compliant implementation. Search Console reports can identify parsing issues, while log file analysis shows which crawlers actually requested the site and which paths they accessed.