Robots.txt issue - site resubmission needed?
-
We recently had an issue when a load of new files were transferred from our dev server to the live site, which unfortunately included the dev site's robots.txt file which had a disallow:/ instruction. Bad!
Luckily I spotted it quickly and the file has been replaced. The extent of the damage seems to be that some descriptions aren't displaying and we're getting a message about robots.txt in the SERPs for a few keywords. I've done a site: search and generally it seems to be OK for 99% of our pages.
Our positions don't seem to be affected right now but obviously it's not great for the CTRs on those keywords affected.
My question is whether there is anything I can do to bring the updated robots.txt file to Google's attention? Or should we just wait and sit it out?
Thanks in advance for your answers!
-
Hi Greg - I've done this and resubmitted the sitemap, but I'm now getting severe health warnings saying that robots.txt is blocking important pages.
I've run our robots.txt file through the tester in Webmaster tools and it is saying allowed, however I'm obviously concerned about the warnings.
Have you experience of anything like this?
Thanks
Rory -
I agree with Michael.
I have also seen a wordpress site that had blocked the robots from the entire site for 1 week.
After allowing the robots back in, we saw the rankings improve with in a few days.
Don't stress, just resubmit the sitemap or create a new one with the effected URL's
Greg
-
Hi,
We had a vaguely similar thing happen where we took over the running over a site and the old dev added a robots.txt disallowing the site from being indexed, which wasn't pick up for a while as dev work was on-going. Basically the sites rankings panned out.
We fixed the issue an re-submitted the site through webmaster tools. Site was reindexed within a week or so and rankings came back over the next 6 weeks.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blocking in Robots.txt and the re-indexing - DA effects?
I have two good high level DA sites that target the US (.com) and UK (.co.uk). The .com ranks well but is dormant from a commercial aspect - the .co.uk is the commercial focus and gets great traffic. Issue is the .com ranks for brand in the UK - I want the .co.uk to rank for brand in the UK. I can't 301 the .com as it will be used again in the near future. I want to block the .com in Robots.txt with a view to un-block it again when I need it. I don't think the DA would be affected as the links stay and the sites live (just not indexed) so when I unblock it should be fine - HOWEVER - my query is things like organic CTR data that Google records and other factors won't contribute to its value. Has anyone ever blocked and un-blocked and whats the affects pls? All answers greatly received - cheers GB
Technical SEO | | Bush_JSM0 -
Blocking subdomains with Robots.txt file
We noticed that Google is indexing our pre-production site ibweb.prod.interstatebatteries.com in addition to indexing our main site interstatebatteries.com. Can you all help shed some light on the proper way to no-index our pre-prod site without impacting our live site?
Technical SEO | | paulwatley0 -
Our site recently switched from http to https. Do I still need to setup a redirect for the incoming links pointing to http?
Our site recently switched from http to https. If you type in the http://www.websitename.com then it will automatically go to https://www.websitename.com ... my question is... do I still need to create a redirect in the htaccess file to ensure we don't lose all the links currently pointing to the http version of the website?
Technical SEO | | ninel_P0 -
I am having an issue with my rankings
I am having an issue with my rankings. I am not sure if there are issues with onpage dup content or with the way wordpress is behaving but there is no reason based upon the sites back link profile that the site shouldn't be ranking well. The site is mesocare.org. If anyone can help it would be appreciated.
Technical SEO | | weitzluxenberg0 -
Need joomla help. site rankings dropped since upgrade
Hi, really having problems here with www.in2town.co.uk our site was always in the top ten for a large number of important keywords and since our upgrade from joomla 1.5 a month ago to joomla 3.0 we have vanished out of a lot of the search engines for important keywords I am concerned that we could be doing something wrong. two of our important keywords is lifestyle magazine and also gastric band hypnotherapy, but we are nowhere to be seen for these keywords. I must be doing something wrong. The site is an old site, we have been around for many years and have always ranked well up until now. We use K2 and i am just wondering if we have set up things wrong. for example under this article it comes under the category of trip advisor http://www.in2town.co.uk/trip-advisor/t ... -term-stay now when you delete the article name and just keep the trip advisor you get a list of the articles and i am wondering if this is damaging my site. http://www.in2town.co.uk/trip-advisor any help in helping me to understand why my site has dropped in rankings would be great.
Technical SEO | | ClaireH-1848860 -
Two sites
Hi there just joined had nightmere of a time trying to get a website up and running..... now i have 2 .... one marketing person did and one i did the one i did performing better on google but other onre looks more profetional is there a way i can conbine the 2 under one site..... the one that looks better and getting the benifit of the one thats performing better...... Thanks steve......
Technical SEO | | stevetemple0 -
Do I need an XML sitemap?
I have an established website that ranks well in Google. However, I have just noticed that no xml sitemap has been registered in Google webmaster tools, so the likelihood is that it hasn't been registered with the other search engines. However, there is an html sitemap listed on the website. Seeing as the website is already ranking well, do I still need to generate and submit an XML sitemap? Could there be any detriment to current rankings in doing so?
Technical SEO | | pugh0 -
I am Posting an article on my site and another site has asked to use the same article - Is this a duplicate content issue with google if i am the creator of the content and will it penalize our sites - or one more than the other??
I operate an ecommerce site for outdoor gear and was invited to guest post on a popular blog (not my site) for a trip i had been on. I wrote the aritcle for them and i also will post this same article on my website. Is this a dup content problem with google? and or the other site? Any Help. Also if i wanted to post this same article to 1 or 2 other blogs as long as they link back to me as the author of the article
Technical SEO | | isle_surf0