To block non-essential pages from being indexed by search engines, using noindex tags is the recommended approach rather than relying on robots.txt. The noindex directive can be implemented either as a <meta> tag in the HTML <head> section or as an HTTP response header (X-Robots-Tag). This tells search engines that support noindex (like Google) not to include those pages in their search results.
Key points for effective use:
-
Do not block pages with robots.txt if you want to use
noindex, because if a page is disallowed in robots.txt, search engines cannot crawl it to see thenoindextag, making the directive ineffective. -
To prevent indexing, add this meta tag inside the
<head>of the page:<meta name="robots" content="noindex"> -
Alternatively, use the HTTP header for non-HTML files or server-side control:
X-Robots-Tag: noindex -
You can combine
noindexwithnofollowif you want to prevent indexing and also prevent crawling of links on the page:<meta name="robots" content="noindex, nofollow"> -
Robots.txt is useful for blocking crawling but not for blocking indexing. If a page is blocked by robots.txt but linked elsewhere, it might still appear in search results without a snippet, which is often undesirable.
In summary, to block non-essential pages from appearing in search results, ensure those pages are crawlable and include a noindex directive either via meta tags or HTTP headers. Avoid disallowing these pages in robots.txt if you want the noindex to work properly.
Maple Ranking offers the highest quality website traffic services in Canada. We provide a variety of traffic services for our clients, including website traffic, desktop traffic, mobile traffic, Google traffic, search traffic, eCommerce traffic, YouTube traffic, and TikTok traffic. Our website boasts a 100% customer satisfaction rate, so you can confidently purchase large amounts of SEO traffic online. For just 720 PHP per month, you can immediately increase website traffic, improve SEO performance, and boost sales!
Having trouble choosing a traffic package? Contact us, and our staff will assist you.
Free consultation