What can I do if Google Webmaster Tools doesn't recognize the robots.txt file?
-
I'm working on a recently hacked site for a client and and in trying to identify how exactly the hack is running I need to use the fetch as Google bot feature in GWT.
I'd love to use this but it thinks the robots.txt is blocking it's acces but the only thing in the robots.txt file is a link to the sitemap.
Unde the Blocked URLs section of the GWT it shows that the robots.txt was last downloaded yesterday but it's incorrect information. Is there a way to force Google to look again?
-
No, but they might write to it, modify it, or do all sorts of other nasty stuff I've seen hackers do when they get a hold of any writeable file on a system.
-
lol it's a robots text file. what are they going to do. Steal it?
I should have clarified do a 777 to make sure that is not your problem, then yes change the permission to be tighter -
Eesh I don't recommend 777. 644 or, if you're going to change it right back, 755 at most.
-
File permission maybe? Change it to 777 and try it again
-
If you have shell access on Linux you can use wget or GET or run lynx.
If google is getting the wrong robots file then your web server must be sending out something other than what you think is the robots file.
What happens if you do this in your browser:
-
Looking in my log files, Google hits robots.txt just about every time it crawls our site.
What are you trying to accomplish using fetch as Googlebot? Any chance CURL could do the job for you, or another tool that ignores robots.txt?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I Block https URLs using Host directive in robots.txt?
Hello Moz Community, Recently, I have found that Google bots has started crawling HTTPs urls of my website which is increasing the number of duplicate pages at our website. Instead of creating a separate robots.txt file for https version of my website, can I use Host directive in the robots.txt to suggest Google bots which is the original version of the website. Host: http://www.example.com I was wondering if this method will work and suggest Google bots that HTTPs URLs are the mirror of this website. Thanks for all of the great responses! Regards,
Technical SEO | | TJC.co.uk
Ramendra0 -
Google how deal with licensed content when this placed on vendor & client's website too. Will Google penalize the client's site for this ?
One of my client bought licensed content from top vendor of Health Industry. This same content is on the vendor's website & my client's site also but on my site there is a link back to vendor is placed which clearly tells to anyone that this is a licensed content & we bought from this vendor. My client bought paid top quality content for best source of industry but at this same this is placed on vendor's website also. Will Google penalize my client's website for this ? Niche is HEALTH
Technical SEO | | sourabhrana1 -
Blocked jquery in Robots.txt, Any SEO impact?
I've heard that Google is now indexing links and stuff available in javascript and jquery. My webmastertools is showing that some links are blocked in robots.txt of jquery. Sorry I'm not a developer or designer. I want to know is there any impact of this on my SEO? and also how can I unblock it for the robots? Check this screenshot: http://i.imgur.com/3VDWikC.png
Technical SEO | | hammadrafique0 -
Adding multi-language sitemaps to robots.txt
I am working on a revamped multi-language site that has moved to Magento. Each language runs off the core coding so there are no sub-directories per language. The developer has created sitemaps which have been uploaded to their respective GWT accounts. They have placed the sitemaps in new directories such as: /sitemap/uk/sitemap.xml /sitemap/de/sitemap.xml I want to add the sitemaps to the robots.txt but can't figure out how to do it. Also should they have placed the sitemaps in a single location with the file identifying each language: /sitemap/uk-sitemap.xml /sitemap/de-sitemap.xml What is the cleanest way of handling these sitemaps and can/should I get them on robots.txt?
Technical SEO | | MickEdwards0 -
Robots.txt best practices & tips
Hey, I was wondering if someone could give me some advice on whether I should block the robots.txt file from the average user (not from googlebot, yandex, etc)? If so, how would I go about doing this? With .htaccess I'm guessing - but not an expert. What can people do with the information in the file? Maybe someone can give me some "best practices"? (I have a wordpress based website) Thanks in advance!
Technical SEO | | JonathanRolande0 -
Suggested crawl rate in google webmaster tools?
hey moz peeps, got a general question: what is the suggested custom crawl rate in google webmaster tools? or is it better to "Let Google determine my crawl rate (recommended)" If you guys have any good suggestions on this and site why that would be very helpful, thanks guys!
Technical SEO | | david3050 -
Is it terrible to not have robots.txt ?
I was under the impression that you really should have a robots.txt page, and not having one is pretty bad. However, hubspot (which I'm not impressed with) does not have the capability of properly implementing one. Will this hurt the site?
Technical SEO | | StandUpCubicles1 -
Can Google read text in Javascript?
We have just completed the redesign of our product page, which you can see here: http://www.uksoccershop.com/p-19045/2011-12-Chelsea-Adidas-Away-Football-Shirt.html Because we want the select size / add to basket section to appear prominently, you can see we are showing only a snippet of the product description in this section and then user has to click "more" to see it. My question is, can Google read the product description here since it's in Javascript? The code is as follows: 2011-12 Chelsea Adidas Away Football Shirt £44.99 Item Code:379606 Brand new, official Chelsea away shirt for the 2011/12 Premiership season, available to buy in adult sizes S, M, L, XL, XXL, XXXL. This football shirt is manufactured by Adidas and is black in colour.[ More...](javascript:void(0);) Brand new, official Chelsea away shirt for the 2011/12 Premiership season, available to buy in adult sizes S, M, L, XL, XXL, XXXL. This football shirt is manufactured by Adidas and is black in colour. Cheer on the Blues in style in the new adidas Chelsea Away Shirt, featuring a striking blue blocked design on an imposing black background complete with the club crest and adidas logo embroidery across the chest for a great style on or off the pitch. The new Chelsea Away Shirt is designed with adidas' ClimaCool technology to bring moisture away from your skin, keeping you cool, comfortable and performing at your best as you emulate the skills of Frank Lampard, Fernando Torres and John Terry on the pitch. Customise your shirt with Premiership shirt printing for your favourite Chelsea stars or choose your own custom name and number. Adult Football Shirt
Technical SEO | | ukss1984
Short sleeves soccer jersey
Chelsea club crest to left chest
adidas logo and stripes
Print sponsor to centre
ClimaCool technology
Machine washable Product code: 379606 The 2011/12 Chelsea away football kit is released on 7th July 2011. <form name="currenychange" action="http://www.uksoccershop.com/p-19045/2011-12-Chelsea-Adidas-Away-Football-Shirt.html" method="get">
<select class="topselectbox" onchange="this.form.submit();" name="currency" style="float:right;"> <option value="USD">US Dollars</option> <option value="EUR">Euro</option> <option value="GBP" selected="selected">UK Sterling</option> <option value="AUD">Australian Dollars</option> </select>
</form> Available Now [Be the first to ask a question](javascript:void(0); "Ask a Question")
[Be the first to review this product](javascript://) Rating: 5 out of 5 stars <form name="cart_quantity" action="http://www.uksoccershop.com/p-19045/2011-12-Chelsea-Adidas-Away-Football-Shirt.html?number_of_uploads=0&action=add_product" method="post" enctype="multipart/form-data"> Which parts of this is Google going to be able to read? Should we make the product title our H1 header for this page and can it currently read that within the code above? </form>0