Andrew Fletcher published: 16 December 2021 1 minute read
Have you noticed through your Google account that there are items being indexed that shouldn't have been? One for me was taxonomy terms. Not important to be followed.
- Being indexed on my sitemap (do I just erase those entries on the XML file?)
- Being crawled (I am guessing this is with a robots.txt file though I have never created one before)
- Being viewed (stumbled upon) -> if this even possible to block?
Resolve these quickly by simply adding disallow in robots.txt file.
Disallow: /taxonomy/
And add no follow no index on taxonomy pages using the Metatags module (/admin/config/search/metatag)
<meta name="robots" content="noindex, nofollow" />
Clear your cache, rebuild your sitemap and cross check that they are gone.