Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Marking a Business as Permanently Closed in GMB Impact Website Rankings?
I recently acquired a business that had a website with great rankings and traffic but also had a few physical locations listed in Google. The physical locations closed and are marked as "Permanently Closed" in Google. The website also expired for several months before I picked it up. The site had amazing rankings and traffic previously but is really struggling to regain rankings now. I'm concerned that the site is not performing well due to the business marked as "permanently closed" in GMB. I'm unable to do anything about that because the website doesn't have any physical locations. Any tips on what to do here?
Technical SEO | | shags3130 -
To merge website and blog?
I was hoping to get a bit of advice if possible. Our website has a domain authority of 25. Our blog, on a separate platform has a domain authority of 73. In essence, we were wondering if we should move all the content from our blog to our website and then set up redirects? What's stopped us so far is the fear that the links from the blog, with the higher domain authority, are having a positive effect on the website, and if we were to move the content we'd lose the value of those backlinks. So I was wondering if you could possibly advise me on the matter? Any help would be much appreciated. Thanks
Technical SEO | | Stone_Junction0 -
Using the Google Remove URL Tool to remove https pages
I have found a way to get a list of 'some' of my 180,000+ garbage URLs now, and I'm going through the tedious task of using the URL removal tool to put them in one at a time. Between that and my robots.txt file and the URL Parameters, I'm hoping to see some change each week. I have noticed when I put URL's starting with https:// in to the removal tool, it adds the http:// main URL at the front. For example, I add to the removal tool:- https://www.mydomain.com/blah.html?search_garbage_url_addition On the confirmation page, the URL actually shows as:- http://www.mydomain.com/https://www.mydomain.com/blah.html?search_garbage_url_addition I don't want to accidentally remove my main URL or cause problems. Is this the right way this should look? AND PART 2 OF MY QUESTION If you see the search description in Google for a page you want removed that says the following in the SERP results, should I still go to the trouble of putting in the removal request? www.domain.com/url.html?xsearch_... A description for this result is not available because of this site's robots.txt – learn more.
Technical SEO | | sparrowdog1 -
Blog separate from Website
One of my clients has a well established website, and a well established blog - each with its own domain. Is there any way to move the blog to his website domain without losing the SEO and links that he has built up over time?
Technical SEO | | EchelonSEO0 -
The use of robots.txt
Could someone please confirm that if I do not want to block any pages from my URL, then I do not need a robots.txt file on my site? Thanks
Technical SEO | | ICON_Malta0 -
Website IP Location
My main target audience is in the UK, but my website's IP is in the United States. Would it be worthwhile to change the IP to a UK address? How would I go about that? Thanks!
Technical SEO | | theLotter0 -
How to attach at text to image that other websites use from my website
I often have other websites link to my website. They will do this with an image that they pull off of my website. (actually my website continues to serve the image). These inbound links are great, but they don't have alt text. Is there a way for me to attach alt text to the images, or is this something the other website needs to code themselves?
Technical SEO | | EugeneF0 -
How does a mega drop-down affects SEO?
We are looking at implementing a "mega drop-down" as our main menu on our website. Will that be good or bad for SEO? My company is a big tour operator so our website contains a lot of pages describing all our destinations, hotels etc. We have noticed that our visitors have some trouble to navigate to all this pages since it requires a lot of clicks to reach a specific page. In order to make this easier we have looked at this popular mega drop-down thing that we all love. But what about Google? Will Google love or hate us for doing this? An example showing what I mean by mega drop-down: http://www.phonehouse.se/
Technical SEO | | Googleankan0