Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Inspection of the website is required.
Hello, Recently We have a taken a project " http://customerconnect-services.com/ " and are responsible for the Digital Marketing for the website. FYI, we have been following the best On-Page SEO practices ever since we took the project: Performing Keyword Research, Finalizing the Keywords,Using those in the Page-Title, Meta-Descriptions, Heading Tags and of course in the Content as per MOZ suggestions & SEO standards. But, we are unable to rank in first page; which is a serious matter of concern for us. We have also checked whether the Domain / URL has been blacklisted, but it's not (Not even by Google). We are therefore unable to figure out what is going wrong even after following so many best practices to get the keywords a good ranking (1st & 2nd page of SERPs). Therefore I would like to request you to provide your expert opinions in this regard by checking what is it that we are not getting right. The website url is http://customerconnect-services.com/. As this is a high-priority issue for us & the client is a prestigious one of course, please help. Looking forward to hearing from you at the earliest. Thanks & Regards,
Technical SEO | | Harini.M
Harini0 -
Robots.txt on subdomains
Hi guys! I keep reading conflicting information on this and it's left me a little unsure. Am I right in thinking that a website with a subdomain of shop.sitetitle.com will share the same robots.txt file as the root domain?
Technical SEO | | Whittie0 -
Website not crawled
i added website www.nsale.in in add campaign, it shows only 1 page crawled. but its working fine for other sites, any idea why it failed ?
Technical SEO | | Dhinesh0 -
Robots.txt and joomla
Hello, I use joomla for my website and automatically all those files are blocked is that good or bad, so I remove anything and if so why ? User-agent: *
Technical SEO | | seoanalytics
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/ I also added to my robots.txt files my email address ( is that useful, I am afraid google passes PR to the email address )
and a javascript: void (0) because I have tabs on my webpage ( is that useful )
as well as a .pdf ( is it also useful ) any comments ? does anything need to be changed or is it ok ? Thank you,0 -
When will seomoz rank my website?
Hi all, I am really excited to be here. I joined seo moz three weeks ago. Seo moz crawled my website twice and still under competitive domain analysis my domain authority is 1 and everything else underneath is zero. I fixed some errors and warning after first crawl but still nothing in root domain metrics. Can anyone please explain whats happening? Thank you!
Technical SEO | | proaspirant0 -
Transfer of website to maintain SEO
I am nearly finished building a new wordpress website to replace the old static HTML site which is currently doing very well in SERPs. My old site is currently hosted by a company who owns the site, my domain is registered with a different company and I have the nameservers pointed at Cloudflare. I have built my new site and have it hosted with a different company and I wish to keep the domain name registered with the company it is currently with.Is there an order that I should follow to move from the old site to the new site using my current domain name so that I minimize any impact on SEO. I built the new site with the same URL structure to maintain linking benefit. thanks
Technical SEO | | casper4340 -
Seo template for new website
I am revamping my website (www.UltimateBasicTraining.com). It's going to be major and I am concerned about the potential traffic loss since over 60% of my overall traffic comes from organic search results in the military basic training area. Are there any good SEO website templates I can start from? I see a lot out there but would prefer the advice of professionals. Thanks
Technical SEO | | TheVolkinator0