Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google: What factors contribute to rank a landing page in a specific country?
Hi community, I would like to know what is the criteria to rank a landing page in specific country Google search. For example, if we want to rank our landing page for "GDPR Australia", what are the factors which will impact beside writing the cotent related to the above mentioned keyword? Thanks
Algorithm Updates | | vtmoz0 -
Hit by an unnamed Google update on November 30th - Still suffering
Hi, Moz Community. Just decided to sign up for a free trial because I'm absolutely at my wits end here. Here's my site: cheapgamesguru.com I run a small PC gaming blog monetized by affiliate marketing. I do all the writing, SEO, etc. myself. The content I write is, from what I can tell, fully complying with Google's guidelines, and in and of itself is pretty informative and high-quality. My site was started in December of 2015, and it was doing very well for a good 10 or 11 months - until late November of 2016. Then something happened. My traffic started plummeting - I went from getting nearly 300 organic users a day (Not sessions - actual unique users) to 80, then 40, and now I'm lucky to get over 15 a day. I do not do ANY black hat SEO whatsoever. I have not taken part in any shady link building schemes, nor do I try to trick Google in any way. I just write good content, do good keyword research (Targeting only low-hanging fruit and low-difficulty keywords using KWFinder), and do my best to provide a good user experience. I run no ads on my site. Glenn Gabe wrote about a potential Google update on November 29th, but the stuff he said in his article doesn't seem to affect me - my mobile site is perfectly fine, according to Google's own metrics and testing tools. Here's the article in question: http://www.gsqi.com/marketing-blog/november-30-2016-google-algorithm-update/ At first, I thought it was possible that this was a result of my competitors simply doing far better than me - but that doesn't seem to be the case, as their rankings did not actually move - mine simply pummeted. And many of their sites are far worse than mine in terms of grammar, spelling, and site speed. I understand backlinks are important, by the way, but I really don't think that's why my site was hit. Many competitors of mine have little to no backlinks and are doing great, and it would also not make much sense for Google to hit an otherwise great site just because they have few backlinks. A friend of mine has reached out to Glenn Gabe himself to see if he can get his input on my site, but he's had a busy schedule and hasn't gotten a chance to take a look yet. I recently obtained a backlink from a highly relevant DA 65 site (About a month ago, same niche as my site), and it now shows up in Search Console and Ahrefs - but it hasn't affected rankings whatsoever. Important Note: I'm not only just ranking poorly for stuff, I'm ranking in position 100-150+ for many low-competition keywords. I have no idea why that is happening - is my site THAT bad, that my content deserves to be ranking on page 15 or lower? Sorry for the long question. I'm struggling here, and just wanted to give as much information as possible. I would really appreciate any input you guys can give me - if any SEO experts want to turn my site into a case study and work with me to improve things, I'd also be open to that 😉 I kid, of course - I know you guys are all busy. Thanks! P.S. I've attached a picture of my SEMRush graph, for reference, as well. mhgSw
Algorithm Updates | | polycountz0 -
Google search analytics position - how is it worked out
In our Google search analytic s graphs total clicks and impressions appear as a sold line on the graph(ie showing a result for each day) Position only shows as an occasional dot or line - not a continuous result for each day) sometimes there are days with no result for position. How do google get these results
Algorithm Updates | | CostumeD0 -
Google Index
Hi all, I just submit my url and linked pages along with xml map to index. How long does it take google to index my new pages?
Algorithm Updates | | businessowner0 -
Fetch as Google in GWT - Functionality
Hi, For example, some of the HTML improvements notices from GWT, says dupe meta descriptions or titles, for pages that have since been 301 redirected or had a canonical tag added. So, my idea is to force google to read it using "Fetch as Google" - hoping that it will now see 301 redirection or the fix we have implemented. Does this work? How long does it take? Lastly, should I just click the "fetch as google" or should I also click on the "Submit to index" button? Thanks!
Algorithm Updates | | bjs20100 -
If Google doesn’t know we’re hosted in the UK, does that affect our SERPs?
Hi, In November 2011 our eCommerce website dropped from between 3rd and 4th position in the UK SERPs down to 7th and 8th. A year after this happened, we still haven’t moved back up to the original ranking despite all our best efforts and we’re looking for a bit of insight into what could have happened. One of our theories is this, do you think it might be the problem? In October 2011 we moved from a single-site custom built CMS hosted in the UK to a multi-site custom built CMS hosted on a much better server based in the UK. As part of this move we started using CloudFlare to help with security and performance (CloudFlare is a security CDN). Because CloudFlare’s servers are in the US, to the outside world it almost looks like we went from a slow hosting company in the UK to a much quicker hosting company in the US. Could this have affected our rankings? We know that Google takes the server IP address into account as a ranking factor, but as far as we understand it’s because they (rightly) believe that a server closer to the user will perform better. So a UK server will serve up pages quicker to a visitor in the UK than a US server because the data has a shorter distance to travel. However, we’re definitely not experiencing an issue with being recognised as a UK website. We have a .co.uk domain (which is obviously a big indicator) and if you click on “Pages from the UK” in the SERPs we jump up to 3rd place. So Google seems to know we’re a UK site. Is the fact we’re using CloudFlare and hence hiding our real server IP address – is this penalising us in the SERPs? Currently out of the 6 websites above us, 4 are in the US and 2 are in the UK. All of these are massive sites with lots of links, so smaller ranking factors might be more important for us. Obviously the big downside of not using CloudFlare is that our site becomes much less secure and it becomes much slower. Images and some static content is distributed via a local CloudFlare server, which means it should tick Google’s box in terms of providing a quick site for users. CloudFlare say in a blog post that they used to have Google crawl rates and geo-tagging issues in the past when they were just starting out, but in 2010 they started working with “the big search engines” to make sure they treated CloudFlare like a CDN (so special rules that apply to Akamai also apply to CloudFlare). Since they’ve been working with Google, CloudFlare say that their customers will only see a positive SEO impact. So at the moment we’re at a loss about what happened to our ranking. Google say they take IP’s into account for ranking, but by using CloudFlare it looks like we’re in the US. We definitely know we’re not having geo-tagging issues and CloudFlare say they’re working with Google to ensure its customers aren't seeing a negative impact by using CloudFlare, but a niggling part of us still wonders whether it could impact our SEO. Many thanks, James
Algorithm Updates | | OptiBacUK0 -
Problems with Google results
Hi Everybody, I ve been dealing with this issue for a while now. i have a multilingual website: www.vallnord.com When a search for Vallnord in Google it always shows the result in Catalan, but it does not show what I specified in the meta description, it displays what it crawls from the home page. I have 2 problems here: It is not showing my meta description. What can I do? It is not showing the language from which the search was made. Example: if you search from Google.com and your default language is english it should been displayed the result from the english HTML. www.vallnord.com/en but it is not like this. It is always the catalan (default language of the site) the one that is displayed. I have tried several things already: Inserting the Hreflang function Changing the descriptions Resubmitting the sitemap via Google Webmaster I can not figure out what is going on because if you search: "Vallnord Castellano" it will display the spanish URL but still not the proper description. Moreover if you search: "www.vallnord.com/es" on google , it will display the proper URL and description. FYI, I am using 301 redirects for the languages: es.vallnord.com it is the sames as www.vallnord.com/es In addition to this, If using Yahoo search engine there is no problem. it will show the proper language. from yahoo.com the first result is in english and from yahoo.es the first result Spanish. So any idea what would be the problem?And furthermore, any Idea which would be the solution? Thanks in advance, Guido.
Algorithm Updates | | SilbertAd0 -
Why would my keywords never ranking in Bing but have great position in both Google and Yahoo?
I have several keywords ranking top 5 for Google and Yahoo but nothing from Bing. Any ideas?
Algorithm Updates | | CIEEwebTeam0