Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Mobile website indexing
Hi we have a mobile version of our website at mobile.gardening-services-edinburgh.com its been live for 5, maybe 6 months, it has its own mobile-sitemap.xml have tried submitting this sitemap to google and for some reason it does not index these pages any ideas, most welcome
Technical SEO | | McSEO0 -
Rankings after manual penalty removal
I've just started working on a ecommerce website that was hit by Penguin 2.0 in May (It was ranking 2nd for it's major keyword at the time) and it hasn't been indexing for that keyword since After a lot of link removal, the reconsideration request was accepted and the manual penalty had been removed. Rankings haven't really improved and that specific keyword has not been reindexed The site does have a lot of not found errors (It was 5.5k but recently taken down to 4k) but it was still ranking before the penalty. Is there anything you believe I'm missing? Is it the onsite errors that are flagging the site as unreliable? I thought it would still appear for the keyword if that was the case
Technical SEO | | Sandeep_Matharu0 -
Robots.txt checker
Google seems to have discontinued their robots.txt checker. Is there another tool that I can use to check my text instead? Thanks!
Technical SEO | | theLotter0 -
Optimizing Flash websites
Does anyone know the latest on how well Google is indexing flash sites? I'm talking to a prospect now whose site is 100% flash, but looking at it all the content within the site is indexed and returning in the results. This is promising enough, but does anyone know exactly how well Google is indexing flash content these days and are there still any problems with that? Thanks!
Technical SEO | | MattBarker0 -
Bad reviews coming next to the company website, how to remove those ??
My website name www.commonsite.com (duplicate name), if i search in google with keyword common site, next to that i'm getting mouthshut bad reviews. I tried various methods till now i didnt get any improvement. Finally my doubt is my site has 6 sitelinks in the search engine. What will happen if i delete that ?? Can i get those pages results next to my main website home page results. Please clarify my doubt about sitelinks.
Technical SEO | | MadhukarSV0 -
Mobile website settings - I am doing right?
Hi, http://www.schicksal.com has a "normal" and a "mobile' version. We are using a browser detection routine to redirect the visitor to the "default site" or the "mobile site". The mobile site is here:
Technical SEO | | GeorgFranz
http://www.schicksal.com/m The robots.txt contains these lines: User-agent: *
Allow: / User-agent: Googlebot
Disallow: /m
Allow: / User-agent: Googlebot-Mobile
Disallow: /
Allow: /m Sitemap: http://www.schicksal.com/sitemaps/index So, the idea is: Only allow the Googlebot-Mobile Bot to access the mobile site. We have also separate sitemaps for default and mobile version. One of the mobile sitemap is here My problem: Webmaster tool is saying that Google received 898 urls from the mobile sitemap, but none has been indexed. (Google has indexed 550 from the "web sitemap".) I've checked the webmaster tools - no errors on the sitemap. So, if you are searching at google.com/m - you are getting results from the default web page, but not the mobile version. This is not that bad because you will be redirected to the mobile version. So, my question: Is this the "normal" behaviour? Or is there something wrong with my config? Would it be better to move the mobile site to a subdomain like m.schicksal.com? Best wishes, Georg.0 -
Mobile Google Not Indexing Mobile Website
Google currently does not index our mobile website. It has the WWW website in it's index. When a user from a mobile phone clicks on a mobile search result for WWW we redirect them to our mobile website. This is posing problems for us as our mobile website is a fraction of the # of pages/sections as our WWW. So for example, mobile search results show that we have a "careers" section; but that's not the case for the mobile website. As a result a user gets a 404. How do we force mobile Google to index our mobile website instead of our WWW?
Technical SEO | | RBA0 -
Removing pages from website
Hello all, I am fairly new to the SEOmoz community. But i am working for a company which organizes exhibitons, events and training in Holland. A lot of these events are only given ones ore twice and then we do not organise them any more because they are no longer relevant. Every event has its own few webpages which provide information about the event and are being indexed by Google. In the past we did not remove any of these events. I was looking in the CMS and saw a lot of events of 2008 and older which are being indexed. To clean the website and the CMS i am thinking of removing these pages of old events. The risk is that these pages have some links to them and are getting some traffic, so if i remove them there is a risk of losing traffic and rankings. What would be the wise thing to do? Make a folder with archive or something? Regards, Ruud
Technical SEO | | RuudHeijnen0