Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to point a framework to your website
Hi, my client use the NJOYN framework to insert all of the new job post on his website. The problem is that this framework is not hosted at the same place that his website is hosted. This create 2 domain name. First: Example.com second: example.njoyn.com How can I tell google that example.noyn.com link with the site example.com? Can I do this by simply 301 redirect the domain from example.njoyn.com to example.com? I know that this is a framework, is there a way to have access to their htaccess file? Finally, is there a way to rewrite example.njoyn.com/job1 to example.com/job1? Thanks alot
Technical SEO | | bigrat950 -
Rankings after manual penalty removal
I've just started working on a ecommerce website that was hit by Penguin 2.0 in May (It was ranking 2nd for it's major keyword at the time) and it hasn't been indexing for that keyword since After a lot of link removal, the reconsideration request was accepted and the manual penalty had been removed. Rankings haven't really improved and that specific keyword has not been reindexed The site does have a lot of not found errors (It was 5.5k but recently taken down to 4k) but it was still ranking before the penalty. Is there anything you believe I'm missing? Is it the onsite errors that are flagging the site as unreliable? I thought it would still appear for the keyword if that was the case
Technical SEO | | Sandeep_Matharu0 -
3 Different Websites but Same Keywords
One of my client targeting same (5 Keywords) for 3 sites. Domain & Web Hosting is same for 3 sites. Site A - 50.72.134.29
Technical SEO | | krishnaxz
Site B - 50.72.140.227
Site C- 50.72.19.70 Some time ago, ranking dropped - but don't know if it is because on above things? Is it OK? What is the best way to target same keywords for 3 different sites.0 -
Is this normal on my website speed tool
Hi, i would like to know if this is normal as i have never come across it before. i have just checked my speed which needs a lot of improving. I use joomla 3.0 and recently had a developer upgrade it from 1.5, but now i am seeing under the speed test my website showing twice which looks like it is causing a time delay. the tool is http://tools.pingdom.com/fpt/#!/dTjwDM/www.in2town.co.uk can someone please look and let me know if this is normal. my site www.in2town.co.uk is coming up twice which seems to be slowing the site down and i have checked this tool with other sites and they are fine many thanks
Technical SEO | | ClaireH-1848860 -
Empty Meta Robots Directive - Harmful?
Hi, We had a coding update and a side-effect of that was that our directive was emptied, in other words it now reads as: on all of the site. I've since noticed that Google's cache date on all of the pages - at least, the ones I tested - have a Cached date of no later than 17 December '12 - that's the Monday after the directive was removed on mass. So, A, does anyone have solid evidence of an empty directive causing problems? Past experience, Matt Cutts, Fishkin quote, etc. And then B - It seems fairly well correlated but, does my entire site's homogenous Cached date point to this tag removal? Or is it fairly normal to have a particular cache date across a large site (we're a large ecommerce site). Our site: http://www.zando.co.za/ I'm having the directive reinstated as soon as Dev permitting. And then, for extra credit, is there a way with Google's API, or perhaps some other tool, to run an arbitrary list and retrieve Cached dates? I'd want to do this for diagnosis purposes and preferably in a way that OK with Google. I'd avoid CURLing for the cached URL and scraping out that dates with BASH, or any such kind of thing. Cheers,
Technical SEO | | RocketZando0 -
Mobile website settings - I am doing right?
Hi, http://www.schicksal.com has a "normal" and a "mobile' version. We are using a browser detection routine to redirect the visitor to the "default site" or the "mobile site". The mobile site is here:
Technical SEO | | GeorgFranz
http://www.schicksal.com/m The robots.txt contains these lines: User-agent: *
Allow: / User-agent: Googlebot
Disallow: /m
Allow: / User-agent: Googlebot-Mobile
Disallow: /
Allow: /m Sitemap: http://www.schicksal.com/sitemaps/index So, the idea is: Only allow the Googlebot-Mobile Bot to access the mobile site. We have also separate sitemaps for default and mobile version. One of the mobile sitemap is here My problem: Webmaster tool is saying that Google received 898 urls from the mobile sitemap, but none has been indexed. (Google has indexed 550 from the "web sitemap".) I've checked the webmaster tools - no errors on the sitemap. So, if you are searching at google.com/m - you are getting results from the default web page, but not the mobile version. This is not that bad because you will be redirected to the mobile version. So, my question: Is this the "normal" behaviour? Or is there something wrong with my config? Would it be better to move the mobile site to a subdomain like m.schicksal.com? Best wishes, Georg.0 -
Rebuilding an old website
Since we have a strong website; meaning high traffic, but we got 2 issues 1. the framework of the design is not user friendly. 2. the current platform is really old; therefor it comes up with technical problems daily/ We are worried about our links which will affect in our new design, what would be wise to do? Thanks
Technical SEO | | apexcue0 -
Robots.txt
Hello Everyone, The problem I'm having is not knowing where to have the robots.txt file on our server. We have our main domain (company.com) with a robots.txt file in the root of the site, but we also have our blog (company.com/blog) where were trying to disallow certain directories from being crawled for SEO purposes... Would having the blog in the sub-directory still need its own robots.txt? or can I reference the directories i don't want crawled within the blog using the root robots.txt file? Thanks for your insight on this matter.
Technical SEO | | BailHotline0