Can URLs blocked with robots.txt hurt your site?
-
We have about 20 testing environments blocked by robots.txt, and these environments contain duplicates of our indexed content. These environments are all blocked by robots.txt, and appearing in google's index as blocked by robots.txt--can they still count against us or hurt us?
I know the best practice to permanently remove these would be to use the noindex tag, but I'm wondering if we leave them they way they are if they can still hurt us.
-
90% not, first of all, check if google indexed them, if not, your robots.txt should do it, however I would reinforce that by making sure those URLs are our of your sitemap file and make sure your robots's disallows are set to ALL *, not just google for example.
Google's duplicity policies are tough, but they will always respect simple policies such as robots.txt.
I had a case in the past when a customer had a dedicated IP, and google somehow found it, so you could see both the domain's pages and IP's pages, both the same, we simply added a .htaccess rule to point the IP requests to the domain, and even when the situation was like that for long, it doesn't seem to have affected them. In theory google penalizes duplicity but not in this particular cases, it is a matter of behavior.
Regards!
-
I've seen people say that in "rare" cases, links blocked by Robots.txt will be shown as search results but there's no way I can imagine that would happen if it's duplicates of your content.
Robots.txt lets a search engine know not to crawl a directory - but if another resource links to it, they may know it exists, just not the content of it. They won't know if it's noindex or not because they don't crawl it - but if they know it exists, they could rarely return it. Duplicate content would have a better result, therefore that better result will be returned, and your test sites should not be...
As far as hurting your site, no way. Unless a page WAS allowed, is duplicate, is now NOT allowed, and hasn't been recrawled. In that case, I can't imagine it would hurt you that much either. I wouldn't worry about it.
(Also, noindex doesn't matter on these pages. At least to Google. Google will see the noindex first and will not crawl the page. Until they crawl the page it doesn't matter if it has one word or 300 directives, they'll never see it. So noindex really wouldn't help unless a page had already slipped through.)
-
I don't believe they are going to hurt you, it is more of a warning that if you are trying to have these indexed that at the moment they can't be accessed. When you don't want them to be indexed i.e. in this case, I don't believe you are suffering because of it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will changing category URLs on site hurt SEO?
Hi Moz Community, We're looking to replace some URLs on our Wordpress site and I want to make sure we won't hurt our SEO with the changes. The site is lushpalm.com When we originally launched our site we created pages (which are linked to in our main menu) to essentially display our categories. We did this as a workaround because we didn’t like the URL to have the word “category” in it. Now we would like to make some changes and we want to make sure we’re not going to hurt our SEO in any way by accidentally duplicating content or otherwise. We want to fix our structure and now link to our category pages from our main menu, BUT we want to change the URL of the category page so that it doesn’t have “category” in it, essentially renaming it the name of the page currently linked to in our main menu. So basically, the category lushpalm.com/category/surf-trips, would be renamed with the URL lushpalm.com/surf-trips and the current page that is at lushpalm.com/surf-trips would be therefore replaced. My questions are: If we did this, would that mean that the previous “lushpalm.com/category/surf-trips” would cease to exist? Or is there some imprint of that out on the web? And if it is then would it re-direct to the new page? Would replacing the current page URL with a category hurt our current SEO in any way? Would this change cause any duplicate pages somehow? Thanks so much for your help!
Intermediate & Advanced SEO | | TaraLP1 -
What is the Redirect Rule for corresponding https urls to new domain with the same https urls?
2 sites have the same urls but the owner wants just the 1 site. So I will be doing a 301 redirect with .htaccess from https://www.example.co.uk/sportsbook/SOCCER/today/ redirecting to https://www.example.com//sportsbook/SOCCER/today/ There are a lot of urls that are the same, so I was wondering what the rule is to put in the file please that will change them all to the corresponding urls? Would this be correct?... RewriteEngine on
Intermediate & Advanced SEO | | WSIDW
RewriteCond %{HTTPS_HOST} ^example.co.uk [NC,OR]
RewriteCond %{HTTPS_HOST} ^www.example.co.uk [NC]
RewriteRule ^(.*)$ https://example.com$1 [L,R=301,NC] Or would a simple rule like this work... redirect 301 / http://www.new domain.com/ If not correct could you please give me the correct rule, thanks! Then of course doing a change of address of address in webmaster tools after. Also... do I still need to do the forwarding from the https://www.example.co.uk/ domain provider after as well? Many thanks for your help in advance.0 -
When the site's entire URL structure changed, should we update the inbound links built pointing to the old URLs?
We're changing our website's URL structures, this means all our site URLs will be changed. After this is done, do we need to update the old inbound external links to point to the new URLs? Yes the old URLs will be 301 redirected to the new URLs too. Many thanks!
Intermediate & Advanced SEO | | Jade1 -
SEO impact when micro site are hosted on third party url
Have a look at this url
Intermediate & Advanced SEO | | kjerstibakke
http://www.bekkevold-as.no/.
On the front page they are providing links to a website created by one of their suppliers. (Three cirkle icons in the middle of the page).
The links make the customers leave the site and open on a different domain, even the branding of the page is for Bekkevold Elektro (see log top right)
The question from me: How will this impact the SEO for Bekkevold Elektro. Would it be better to create a subdomain for Bekkevold Elektro and ask the supplier to point to this? Or is it ok to leave it as it is, having links back to the web site for Bekkevold Elektro? Just changing the target of the link to open in a new window.
Thank you very much for your support on this. I would appreciate any suggestion for improvement for my customer, Bekkevold Elektro. Best regards
Kjersti Bakke0 -
The review stars for my ecommerce site in the organic search disappeared, how can I have them shown again?
We run www.prams.net, an ecommerce store for strollers and other baby products out of the Uk. We always followed schema and the stars appeared in our product snippets in the organic search, helping us a lot. A couple of weeks ago the stars stopped appearing. When I do the site:www.prams.net they appear but not in the search results anymore. We have a french site as well. www.poussette.com, their it still works. Has anybody got tips what we can do to make the stars appear again? Thanks in advance. Dieter Lang
Intermediate & Advanced SEO | | Storesco0 -
Meta robots or robot.txt file?
Hi Mozzers! For parametric URL's would you recommend meta robot or robot.txt file?
Intermediate & Advanced SEO | | eLab_London
For example: http://www.exmaple.com//category/product/cat no./quickView I want to stop indexing /quickView URLs. And what's the real difference between the two? Thanks again! Kay0 -
Block subdomain directory in robots.txt
Instead of block an entire sub-domain (fr.sitegeek.com) with robots.txt, we like to block one directory (fr.sitegeek.com/blog).
Intermediate & Advanced SEO | | gamesecure
'fr.sitegeek.com/blog' and 'wwww.sitegeek.com/blog' contain the same articles in one language only labels are changed for 'fr' version and we suppose that duplicate content cause problem for SEO. We would like to crawl and index 'www.sitegee.com/blog' articles not 'fr.sitegeek.com/blog'. so, suggest us how to block single sub-domain directory (fr.sitegeek.com/blog) with robot.txt? This is only for blog directory of 'fr' version even all other directories or pages would be crawled and indexed for 'fr' version. Thanks,
Rajiv0 -
Can someone please help me understand my sites recent loss of rankings?
My site has been top 3 for 'speed dating' on Google.co.uk since about 2003 and it went to below top 50 for a lot of it's main keywords shortly after 27 Oct 2012. I did a re-submission request and was told there was 'no manual spam action'. I have a Page Authority of 53, a regular blog http://bit.ly/oKyi88, a KLOUT of 40, user reviews and quality content. I did discover that another URL I using was set to a 302 instead of a 301 for some reason. I don't necessarily think this was an issue as Google should know which is the trusted URL and therefore which content to list. I removed this redirect completely about 3 weeks ago, but I've seen no improvement. I'm looking at improving various things, but I'm still not sure why I've been hit and wonder if I'm missing something obvious? Any suggestions greatly appreciated.
Intermediate & Advanced SEO | | benners0