Robots Meta Tag
A robots meta tag is page-level HTML metadata that tells supported search crawlers how a page may be indexed and displayed in search results.
What is a robots meta tag?
A robots meta tag is an HTML element that gives supported search crawlers page-level instructions about indexing and search-result presentation. It normally appears inside the <head> of an HTML page.
A common example is:
<meta name="robots" content="noindex, follow">
The name="robots" value addresses crawlers generally. A site can also address a specific crawler, such as Googlebot, when a supported crawler-specific rule is required.
What can a robots meta tag control?
Robots directives can control whether a page should appear in search results and how parts of its search snippet may be presented. Common directives include:
- index: Allows the page to be considered for indexing. This is normally the default and does not need to be stated.
- noindex: Instructs supported search engines not to show the page in search results.
- follow: Allows links on the page to be followed. This is generally the default.
- nofollow: Signals that links on the page should not be followed.
- nosnippet: Prevents a text snippet or video preview from being shown for the page.
- max-snippet: Sets a maximum length for a text snippet when supported.
- noarchive: Requests that a cached copy not be shown by search engines that support the directive.
Support differs between search engines. Use directives documented by the crawler that matters to the website rather than assuming every bot interprets every value identically.
Robots meta tag vs. robots.txt
A robots meta tag controls indexing and search presentation for an individual page. A robots.txt file controls which URL paths compliant crawlers may request.
This difference is critical. A crawler must access a page to read its robots meta tag. If the same page is blocked in robots.txt, the crawler may never see the noindex instruction. A blocked URL can therefore still appear in search results when external or internal links reveal its address.
Use noindex when a public page should be crawlable but excluded from search. Use robots.txt mainly to manage crawler access and avoid unnecessary requests, not as a dependable method for removing a page from an index.
Robots meta tag vs. canonical tag
A robots meta tag and a canonical tag have different purposes. A canonical tag indicates the preferred version among duplicate or highly similar URLs. A noindex directive asks for a page to be excluded from search results.
Do not combine them automatically. If duplicate pages should consolidate their signals around one representative URL, canonicalisation is usually the relevant mechanism. If a page should remain accessible but should not appear in search, noindex may be appropriate.
When should a page use noindex?
Potential use cases include:
- Internal search-result pages.
- Thank-you and confirmation pages.
- Utility pages without standalone search value.
- Private-looking but publicly accessible campaign steps that contain no sensitive data.
- Duplicate filter views that visitors need but searchers do not.
- Temporary pages that must remain accessible during a controlled transition.
Do not use noindex to protect confidential information. A public URL can still be visited, shared, logged, or accessed by bots that ignore the directive. Sensitive content requires authentication or another genuine access control.
What is an X-Robots-Tag?
An X-Robots-Tag provides robots directives through an HTTP response header rather than HTML. It is useful for non-HTML resources such as PDFs, images, and other files that cannot contain a meta tag.
The robots meta tag is normally simpler for HTML pages. The response header is useful when control must be applied at the server or file-type level.
What are common robots meta tag mistakes?
- Blocking a page in robots.txt and expecting a hidden noindex tag to be processed.
- Applying noindex to an important service, product, or category template.
- Leaving noindex active after a staging page becomes production content.
- Using noindex instead of authentication for sensitive information.
- Combining conflicting page-level and HTTP-header directives.
- Adding directives without checking whether the target crawler supports them.
How do robots directives work in Webflow?
Webflow page settings include controls for excluding pages from site search and search-engine indexing. Teams can also add custom metadata when a more specific directive is needed.
Be careful with CMS templates: a template-level setting can affect every item in the collection. After migrations, launches, and staging changes, verify the rendered HTML and use Search Console URL Inspection for important pages.
Key takeaway
A robots meta tag provides page-level instructions for supported search crawlers. Use it deliberately to control indexing or snippets, keep it consistent with robots.txt and canonical signals, and never treat it as a security control.
Written by:
I am the co-founder of Overflow Agency and a B2B marketing strategist. I help marketing teams turn their websites into scalable growth systems by combining positioning, design, SEO, AI Search and conversion strategy.