Should I robots block this directory?
-
There's about 43k pages indexed in this directory, and while helpful to end users, I don't see it being a great source of unique content for search engines.
Would you robots block or meta noindex nofollow these pages in the /blissindex/ directory?
ie.
http://www.careerbliss.com/blissindex/petsmart-index-980481/
http://www.careerbliss.com/blissindex/att-index-1043730/
http://www.careerbliss.com/blissindex/facebook-index-996632/
-
Totally agree with Ryan Kent. You should write a paragraph of content that is unique to the company featured. The chart is not unique enough and you will get flagged as having a high ratio of duplicate content. You should also look at all the other SEO elements on this page, understand what keyphrases you are targeting and modify the title, meta and H1 tags.
-
Should I robots block this directory?
I wouldn't.
Robots.txt in general should only be used when there is no other alternate means available to block content. An example is when your site is created by a CMS or e-commerce platform which does not offer the flexibility to noindex individual pages.
By blocking your site's content, you are preventing search engines not only from indexing the pages, but from following any links on those pages. You are restricting the way a crawler can travel on your site, which is generally a bad idea.
Additionally, I would suggest those pages offer value. "Petco salary comparison", "Target wages" and other search queries could generate results for those pages. Those pages contain helpful information which is otherwise not easily found on the internet. If that was my site, I would work to improve the optimization of those pages, not block them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to rank if you are an aggregator or a directory of resource?
Most of the SEO suggestions (great quality content, long form content, engagement rate/time on the page, authority inbound links ) apply to content oriented site. But what should you do if you are an aggregator or a resource directory? You aim is to send the user faster to other site they are looking for or provide ranking about the resources. In fact at a very basic level you are competing for search engine traffic because they are doing same things. You may have done a hand crafted, human created resource that is better than what algorithms are showing. And your site likely to have lot more outgoing links than content. You know you are better (or getting better) since repeat visitors keep coming back. So in these days of Search engines, what a resource directory or aggregator site do to rank? Because even directories need first time visitors till they start coming back again.
Intermediate & Advanced SEO | | Maayboli0 -
Directory with Duplicate content? what to do?
Moz keeps finding loads of pages with duplicate content on my website. The problem is its a directory page to different locations. E.g if we were a clothes shop we would be listing our locations: www.sitename.com/locations/london www.sitename.com/locations/rome www.sitename.com/locations/germany The content on these pages is all the same, except for an embedded google map that shows the location of the place. The problem is that google thinks all these pages are duplicated content. Should i set a canonical link on every single page saying that www.sitename.com/locations/london is the main page? I don't know if i can use canonical links because the page content isn't identical because of the embedded map. Help would be appreciated. Thanks.
Intermediate & Advanced SEO | | nchlondon0 -
Whats the best way to set up a directory listing website
Hello all, I am building a website that lists homeschool events and field trips across various states (locker-time.com) and I have a few questions on setting it up correctly. Both the events and field trips are searchable by distance. For clarification, events are associated with a specific date and time and field trips are not. I currently have a link that says homeschool events and you enter your zip to find things close by. Is it better to create a separate page for each state I am targeting instead? So the link would be homeschool events and then a sub-link that says homeschool events in GA and the GA page brings up all the events in GA, still searchable by zip. Or does it matter? I was thinking if its a separate page, I could put keyword rich copy on top, but then clicking on the menu and choosing the appropriate sub-menu is an additional step for users on the site and as the number of states increase, that sub-menu could get pretty big. The search results pages lists the post title of any events or field trips found and the links go to a page on my website with more information, such as the location, details on the event / field trip and a link to their website. I am wondering for SEO purposes, is this the right way to do it? Or I could set up the results page to show an excerpt and some listing info and then link directly to their website. Does it matter? I was thinking a page on my own website since then I could add images (but that might end up sucking up all my hosting space). As I am adding these listings to my website, I simply copied/pasted the details on the event. Now that I'm thinking about it, original content is best, so should I stop doing that and rewrite the description in my own words? Since the events are date specific events and when they pass, they are no longer on the site, does it matter as much for the events? The field trips do not have dates associated with them, so I can probably work on creating my own descriptions for those. Just not sure if I should bother with events that are more short term. Thanks in advance for ANY advice or suggestions. I'm so looking forward to getting this all set up correctly! I find working on this SEO stuff such fun! Jeanette
Intermediate & Advanced SEO | | fatcreat0 -
Google Indexing Duplicate URLs : Ignoring Robots & Canonical Tags
Hi Moz Community, We have the following robots command that should prevent URLs with tracking parameters being indexed. Disallow: /*? We have noticed google has started indexing pages that are using tracking parameters. Example below. http://www.oakfurnitureland.co.uk/furniture/original-rustic-solid-oak-4-drawer-storage-coffee-table/1149.html http://www.oakfurnitureland.co.uk/furniture/original-rustic-solid-oak-4-drawer-storage-coffee-table/1149.html?ec=affee77a60fe4867 These pages are identified as duplicate content yet have the correct canonical tags: https://www.google.co.uk/search?num=100&site=&source=hp&q=site%3Ahttp%3A%2F%2Fwww.oakfurnitureland.co.uk%2Ffurniture%2Foriginal-rustic-solid-oak-4-drawer-storage-coffee-table%2F1149.html&oq=site%3Ahttp%3A%2F%2Fwww.oakfurnitureland.co.uk%2Ffurniture%2Foriginal-rustic-solid-oak-4-drawer-storage-coffee-table%2F1149.html&gs_l=hp.3..0i10j0l9.4201.5461.0.5879.8.8.0.0.0.0.82.376.7.7.0....0...1c.1.58.hp..3.5.268.0.JTW91YEkjh4 With various affiliate feeds available for our site, we effectively have duplicate versions of every page due to the tracking query that Google seems to be willing to index, ignoring both robots rules & canonical tags. Can anyone shed any light onto the situation?
Intermediate & Advanced SEO | | JBGlobalSEO0 -
Any reason not to redirect entire directory from old site structure to new?
I'm helping on a site that has tons of content and recently moved from a 10 year old .ASP structure to WordPress. There are ~800 404s, with 99% of them in the same directory that is no longer used at all. The old URL structures offer no indication of what the old page contents was. So, there is basically no way to manually redirect page by page to the new site at this point.....is there any reason not to redirect that entire old directory to the new homepage? Matt Cutts seems to think its OK to point an entire old directory to a new homepage, but its not as good as the 1:1 redirects: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93633 Any thoughts?
Intermediate & Advanced SEO | | wattssw0 -
New server update + wrong robots.txt = lost SERP rankings
Over the weekend, we updated our store to a new server. Before the switch, we had a robots.txt file on the new server that disallowed its contents from being indexed (we didn't want duplicate pages from both old and new servers). When we finally made the switch, we somehow forgot to remove that robots.txt file, so the new pages weren't indexed. We quickly put our good robots.txt in place, and we submitted a request for a re-crawl of the site. The problem is that many of our search rankings have changed. We were ranking #2 for some keywords, and now we're not showing up at all. Is there anything we can do? Google Webmaster Tools says that the next crawl could take up to weeks! Any suggestions will be much appreciated.
Intermediate & Advanced SEO | | 9Studios0 -
Redirecting www.example.com to www.example.com/directory/
Hi All, There's been some internal debate going back and forth about redirecting the homepage of a site to a directory. There are a few different POVs circulating, one of which is that it's no different than redirecting to a /index page. Basically, the homepage is ranking for the keyword that we want the directory to rank for but I can't seem to justify placing this type of redirect. The content on both pages is different, but for the term both the homepage and the directory make sense to rank. Has anyone ever done anything like this before? Can anyone see any reason to do something like this? I believe this move would dilute the link value we currently have going to the homepage and potentially cause us to lose our #2 slot with the homepage in favor of a lower spot with the directory. I'd love to hear any thoughts on this/learn if anyone has experimented with this tactic. Thanks in advance!
Intermediate & Advanced SEO | | JamieCottle280 -
Not using a robot command meta tag
Hi SEOmoz peeps. Was doing some research on robot commands and found a couple major sites that are not using them. If you check out the code for these: http://www.amazon.com http://www.zappos.com http://www.zappos.com/product/7787787/color/92100 http://www.altrec.com/ You fill not find a meta robot command line. Of course you need the line for any noindex, nofollow, noarchive pages. However for pages you want crawled and indexed, is there any benefit for not having the line at all? Thanks!
Intermediate & Advanced SEO | | STPseo0