Should I robots block this directory?
-
There's about 43k pages indexed in this directory, and while helpful to end users, I don't see it being a great source of unique content for search engines.
Would you robots block or meta noindex nofollow these pages in the /blissindex/ directory?
ie.
http://www.careerbliss.com/blissindex/petsmart-index-980481/
http://www.careerbliss.com/blissindex/att-index-1043730/
http://www.careerbliss.com/blissindex/facebook-index-996632/
-
Totally agree with Ryan Kent. You should write a paragraph of content that is unique to the company featured. The chart is not unique enough and you will get flagged as having a high ratio of duplicate content. You should also look at all the other SEO elements on this page, understand what keyphrases you are targeting and modify the title, meta and H1 tags.
-
Should I robots block this directory?
I wouldn't.
Robots.txt in general should only be used when there is no other alternate means available to block content. An example is when your site is created by a CMS or e-commerce platform which does not offer the flexibility to noindex individual pages.
By blocking your site's content, you are preventing search engines not only from indexing the pages, but from following any links on those pages. You are restricting the way a crawler can travel on your site, which is generally a bad idea.
Additionally, I would suggest those pages offer value. "Petco salary comparison", "Target wages" and other search queries could generate results for those pages. Those pages contain helpful information which is otherwise not easily found on the internet. If that was my site, I would work to improve the optimization of those pages, not block them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Would you pursue this paid directory link?
Hi all, I KNOW the hard and true answer to this, but I'm looking for deeper insights regarding Links like those contained on this page. I understand the by-the-book answer to this would be only pursue a paid link if it is "nofollowed" OR if it has the potential to bring in new business and traffic. My question is ....does a link like this actually pass SEO value? I see businesses killing it from an SEO standpoint with link profiles full of paid directory links like this. I also thought this conversation was more interested now that Google appears to devaluing links like this instead of issue penalties. Thoughts??
Intermediate & Advanced SEO | | RickyShockley0 -
Robots.txt advice
Hey Guys, Have you ever seen coding like this in a robots.txt, I have never seen a noindex rule in a robots.txt file before - have you? user-agent: AhrefsBot User-agent: trovitBot
Intermediate & Advanced SEO | | eLab_London
User-agent: Nutch
User-agent: Baiduspider
Disallow: / User-agent: *
Disallow: /WebServices/
Disallow: /*?notfound=
Disallow: /?list=
Noindex: /?*list=
Noindex: /local/
Disallow: /local/
Noindex: /handle/
Disallow: /handle/
Noindex: /Handle/
Disallow: /Handle/
Noindex: /localsites/
Disallow: /localsites/
Noindex: /search/
Disallow: /search/
Noindex: /Search/
Disallow: /Search/
Disallow: ? I have never seen a noindex rule in a robots.txt file before - have you?
Any pointers?0 -
Robots.txt - blocking JavaScript and CSS, best practice for Magento
Hi Mozzers, I'm looking for some feedback regarding best practices for setting up Robots.txt file in Magento. I'm concerned we are blocking bots from crawling essential information for page rank. My main concern comes with blocking JavaScript and CSS, are you supposed to block JavaScript and CSS or not? You can view our robots.txt file here Thanks, Blake
Intermediate & Advanced SEO | | LeapOfBelief0 -
Robots.txt & Duplicate Content
In reviewing my crawl results I have 5666 pages of duplicate content. I believe this is because many of the indexed pages are just different ways to get to the same content. There is one primary culprit. It's a series of URL's related to CatalogSearch - for example; http://www.careerbags.com/catalogsearch/result/index/?q=Mobile I have 10074 of those links indexed according to my MOZ crawl. Of those 5349 are tagged as duplicate content. Another 4725 are not. Here are some additional sample links: http://www.careerbags.com/catalogsearch/result/index/?dir=desc&order=relevance&p=2&q=Amy
Intermediate & Advanced SEO | | Careerbags
http://www.careerbags.com/catalogsearch/result/index/?color=28&q=bellemonde
http://www.careerbags.com/catalogsearch/result/index/?cat=9&color=241&dir=asc&order=relevance&q=baggallini All of these links are just different ways of searching through our product catalog. My question is should we disallow - catalogsearch via the robots file? Are these links doing more harm than good?0 -
I tried the directorie list of seomoz, but almost all of them charged for the inclusion. This is a black hat situation?
I need backlinks for my site, and several places inform that directories are a good place. But they charge for the inclusion. Should I pay? This is a blackhat situation where I'm buying for links?
Intermediate & Advanced SEO | | Naghirniac0 -
Negative impact on crawling after upload robots.txt file on HTTPS pages
I experienced negative impact on crawling after upload robots.txt file on HTTPS pages. You can find out both URLs as follow. Robots.txt File for HTTP: http://www.vistastores.com/robots.txt Robots.txt File for HTTPS: https://www.vistastores.com/robots.txt I have disallowed all crawlers for HTTPS pages with following syntax. User-agent: *
Intermediate & Advanced SEO | | CommercePundit
Disallow: / Does it matter for that? If I have done any thing wrong so give me more idea to fix this issue.0 -
Category Pages - Canonical, Robots.txt, Changing Page Attributes
A site has category pages as such: www.domain.com/category.html, www.domain.com/category-page2.html, etc... This is producing duplicate meta descriptions (page titles have page numbers in them so they are not duplicate). Below are the options that we've been thinking about: a. Keep meta descriptions the same except for adding a page number (this would keep internal juice flowing to products that are listed on subsequent pages). All pages have unique product listings. b. Use canonical tags on subsequent pages and point them back to the main category page. c. Robots.txt on subsequent pages. d. ? Options b and c will orphan or french fry some of our product pages. Any help on this would be much appreciated. Thank you.
Intermediate & Advanced SEO | | Troyville0 -
Should we block urls like this - domainname/shop/leather-chairs.html?brand=244&cat=16&dir=ascℴ=price&price=1 within the robots.txt?
I've recently added a campaign within the SEOmoz interface and received an alarming number of errors ~9,000 on our eCommerce website. This site was built in Magento, and we are using search friendly url's however most of our errors were duplicate content / titles due to url's like: domainname/shop/leather-chairs.html?brand=244&cat=16&dir=asc&order=price&price=1 and domainname/shop/leather-chairs.html?brand=244&cat=16&dir=asc&order=price&price=4. Is this hurting us in the search engines? Is rogerbot too good? What can we do to cut off bots after the ".html?" ? Any help would be much appreciated 🙂
Intermediate & Advanced SEO | | MonsterWeb280