How to Disallow Tag Pages With Robot.txt
-
Hi i have a site which i'm dealing with that has tag pages for instant -
http://www.domain.com/news/?tag=choice
How can i exclude these tag pages (about 20+ being crawled and indexed by the search engines with robot.txt
Also sometimes they're created dynamically so i want something which automatically excludes tage pages from being crawled and indexed.
Any suggestions?
Cheers,
Mark
-
Hi Nakul, its Drupal
Mark
-
What CMS is it Mark ?
-
Thanks, is there a way to test it out before actually implementing it with the site.
The site is non-wordpress aswell.
Cheers,
Mark
-
I agree. I would suggest adding the noindex on the pages and letting the bots crawl them. Blocking them would prevent future crawl of these pages, but I am guessing you would also want to remove the existing pages.
Therefore add the noindex first, wait a few days and then add the disallow (Although technically if they are noindex, you don't really need the disallow).
-
Hi Mark
If your using Wordpress then I would recommend SEO Yoast to resolve the tag issue. If not then I suggest you amend the robots.txt file to resolve.
Here is an example:
Disallow: /?tag=
Disallow: /?subcats=
Disallow: /*?features_hash=NOTE:
Be very careful when blocking search engines. Test and test again!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I put rel next and rel prev and canonical on tags pages
Hi I have a tag pages on a news website each tag page is divided to several pages, but Google does't crawled those pages because the links are in javaScript, I want to do the following things: Change the links to html href Add rel=pref rel=next Add a canonical in each page with the url of the main tag page Do you agree with my solution? Thanks Roy
Intermediate & Advanced SEO | | kadut1 -
Multiple pages optimised for the same keywords but pages are functionally different and visually different
Hi MOZ community! We're wondering what the implications would be on organic ranking by having 2 pages, which have quite different functionality were optimised for the same keywords. So, for example, one of the pages in question is
Intermediate & Advanced SEO | | TrueluxGroup
https://www.whichledlight.com/categories/led-spotlights
and the other page is
https://www.whichledlight.com/t/led-spotlights both of these pages are basically geared towards the keyword led spotlights the first link essentially shows the options for led spotlights, the different kind of fittings available, and the second link is a product search / results page for all products that are spotlights. We're wondering what the implications of this could be, as we are currently looking to improve the ranking for the site particularly for this keyword. Is this even safe to do? Especially since we're at the bottom of the hill of climbing the ranking ladder of this keyword. Give us a shout if you want any more detail on this to answer more easily 🙂0 -
Robots txt is case senstive? Pls suggest
Hi i have seen few urls in the html improvements duplicate titles Can i disable one of the below url in the robots.txt? /store/Solar-Home-UPS-1KV-System/75652
Intermediate & Advanced SEO | | Rahim119
/store/solar-home-ups-1kv-system/75652 if i disable this Disallow: /store/Solar-Home-UPS-1KV-System/75652 will the Search engines scan this /store/solar-home-ups-1kv-system/75652 im little confused with case senstive.. Pls suggest go ahead or not in the robots.txt0 -
Should I disallow via robots.txt for my sub folder country TLD's?
Hello, My website is in default English and Spanish as a sub folder TLD. Because of my Joomla platform, Google is listing hundreds of soft 404 links of French, Chinese, German etc. sub TLD's. Again, i never created these country sub folder url's, but Google is crawling them. Is it best to just "Disallow" these sub folder TLD's like the example below, then "mark as fixed" in my crawl errors section in Google Webmaster tools?: User-agent: * Disallow: /de/ Disallow: /fr/ Disallow: /cn/ Thank you, Shawn
Intermediate & Advanced SEO | | Shawn1240 -
Shall i mark my tag pages as nofollow
Ran moz crawl, the tag pages are coming up with missing description Is it okay if the tag been 'noindexed', though they are not coming in as duplicate one Some examples been Gagan Modi - Blog entries tagged in finance
Intermediate & Advanced SEO | | Modi
http://www.mycarhelpline.com/index.php?option=com_easyblog&view=blogger&layout=statistic&id=128&
stat=tag&tagid=67&Itemid=91 Gagan Modi - Blog entries tagged in nissan
http://www.mycarhelpline.com/index.php?option=com_easyblog&view=blogger&layout=statistic&id=128&
stat=tag&tagid=68&Itemid=91 Gagan Modi - Blog entries tagged in dc avanti
http://www.mycarhelpline.com/index.php?option=com_easyblog&view=blogger&layout=statistic&id=128&
stat=tag&tagid=69&Itemid=91 Gagan Modi - Blog entries tagged in mahindra
http://www.mycarhelpline.com/index.php?option=com_easyblog&view=blogger&layout=statistic&id=128&
stat=tag&tagid=7&Itemid=91 Gagan Modi - Blog entries tagged in budget
http://www.mycarhelpline.com/index.php?option=com_easyblog&view=blogger&layout=statistic&id=128&
stat=tag&tagid=72&Itemid=91 Gagan Modi - Blog entries tagged in datsun
http://www.mycarhelpline.com/index.php?option=com_easyblog&view=blogger&layout=statistic&id=128&
stat=tag&tagid=73&Itemid=910 -
Why does SEOmoz bot see duplicate pages despite I am using the canonical tag?
Hello here, today SEOmoz bot found and marked as "duplicate content" the following pages on my website: http://www.virtualsheetmusic.com/score/PatrickCollectionFlPf.html?tab=mp3 http://www.virtualsheetmusic.com/score/PatrickCollectionFlPf.html?tab=pdf And I am wondering why considering the fact I am using on both those pages a canonical tag pointing to the main product page below: http://www.virtualsheetmusic.com/score/PatrickCollectionFlPf.html Shouldn't SEOmoz bot follow the canonical directive and not report those two pages as duplicate? Thank you for any insights I am probably missing here!
Intermediate & Advanced SEO | | fablau0 -
Meta Refresh tag on cache pages- GRRR!
Hi guys, All of our product pages originate in a URL with a unique number but it redirects to an SEO url for the user. These product pages have blocks on the page and these blocks are automatically populated with our database of content. Here's an example of the redirect in place: www.example.com/45643/xxxx.html redirects to www.example.com/seo-friendly-url.html The development team did this for 2 reasons. 1) our internal search needs the unique numbered urls for search and 2) it allows quick redirects as pages are cached. The problem I face is this, the redirects from the cached are being tagged with 'meta refresh', yup, they are 302. The development team said they could stop caching and respond dynamically with a 301 but this would bring in a delay. Speed wise, the cached pages load within 22ms and dynamically 530ms, so yeah half a second more. Currently cached pages just do a meta refresh tagged redirect and I want to move away from this. What would you guys recommend in such a situation? I feel like unless I place a 301, I'll be losing out on rank juice.
Intermediate & Advanced SEO | | Bio-RadAbs0 -
Hide H1 tags on pages. Don't chuckle-Need assistance.
I redesigned my companies website and I am first and foremost an SEO person so I know the importance of a well laid out website. Furthermore, I know realistically you should NEVER hide text whether it's with WH or BH intentions but here is my problem. For every page I have all the details taken care of except proper placement of H1 tags. My website is responsive designed VERY competitive industry I have to make sure it is properly developed both design wise and seo wise It's an INC 5000 company so NO BH intentions On phones and tablet devices I have the header images hidden and in the place of header images I have the information as in location, service,etc of whatever that page may be. This makes it look good on desktops and serves up information quickly to people using phones and tablets. My question is: Would it be bad to turn that text seen on tablets and phones into an h1 tag as it's hidden on desktops with CSS but available on mobile devices. My problem is making the h1 tag's work with the desktop versions visually as placement doesn't make since. Any opinions are appreciated. Thanks Ballanrk
Intermediate & Advanced SEO | | ballanrk0