Definition
Robots Tag
A page-level instruction that guides search engine crawling and indexing behavior.
Definition
A robots tag is metadata, often written as a meta robots tag or HTTP header, that tells crawlers whether to index a page, follow links, or apply preview limits.
Meaning
It controls how search engines should treat a specific page after they discover it.
Why It Matters
Incorrect robots directives can hide important pages from search or expose pages that should not be indexed.
History And Context
Robots directives evolved from the robots.txt convention of the 1990s into page-level controls such as noindex, nofollow, noarchive, and preview directives.