Omitting taxonomies and terms from search engine indexing

[danielfdickinson]

I wonder how folks feel about modifying head.html to have

{{- if and (hugo.IsProduction | or (eq .Site.Params.env "production")) (not .Data.Terms) (not .Data.Pages) }}
<meta name="robots" content="index, follow">
{{- else }}
<meta name="robots" content="noindex, nofollow">
{{- end }}

So that taxonomies and terms do not get indexed by search engines. It seems to me these are not normally 'real' content pages and SEO analysis complains about the lack of description, so it seem preferrable to simply not index them. The lack of description can be dealt with, but it seems to me it's not all that useful to have these pages indexed anyway.

[adityatelange]

robots.txt is made for that https://www.cloudflare.com/learning/bots/what-is-robots.txt/

[adityatelange]

Add it inlayouts/robots.txt

[danielfdickinson]

I don't know why I didn't think of that. Thanks.

[thiyagu-7]

@adityatelange
Aren't robots.txt and robots meta tag supposed to have different purposes?

As per https://developers.google.com/search/docs/crawling-indexing/robots/intro

While Google won't crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google search results. To properly prevent your URL from appearing in Google search results, password-protect the files on your server, use the noindex meta tag or response header, or remove the page entirely.