Common directives used in robots.txt: User-agent, Disallow, Allow, Sitemap

The common directives used in a robots.txt file are:

• User-agent: Specifies which web crawlers (bots) the rules apply to. It can target a specific bot (e.g., Googlebot) or all bots using a wildcard (*).

• Disallow: Tells the specified user-agent which paths or files it is not allowed to crawl. If a path is disallowed, the crawler should not access it.

• Allow: Overrides a disallow directive to permit crawling of a specific path or file within a disallowed directory.

• Sitemap: Provides the location of the website’s sitemap to help crawlers efficiently find and index pages.

These directives form groups in the robots.txt file, where each group starts with a user-agent line followed by allow or disallow rules for that agent. The syntax is case-sensitive for paths but not for directive names. Comments can be added using the # symbol.

For example, a simple group might look like:

User-agent: *
Disallow: /private/
Allow: /private/public-info.html
Sitemap: https://example.com/sitemap.xml

This means all bots are disallowed from crawling the /private/ directory except the public-info.html file, and the sitemap location is provided for better crawling.