Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to track google auto search suggestion click?
Hello Guys, In google.co.uk when I search SEL and google gives me option of different different sites and when I click on any one site then that click tracking I need. I have attached the screenshot to understand easily. Is it possible to track such things or possible via server logs etc? TV99h
Reporting & Analytics | | micey1231 -
Should Google Trends Match Organic Traffic to My Site?
When looking at Google Trends and my Organic Traffic (using GA) as percentages of their total yearly values I have a correlation of .47. This correlation doesn't seem right when you consider that Google Trends (which is showing relative search traffic data) should match up pretty strongly to your Organic Traffic. Any thoughts on what might be going on? Why isn't Google Trends correlating with Organic Traffic? Shouldn't they be pulling from the same data set? Thanks, Jacob
Reporting & Analytics | | jacob.young.cricut0 -
Increase in 404 errors in Webmaster Tools
We have recently updated our website www.cooke.co.uk and twice webmaster tools have reported an increase in 404 errors. However, these errors are not for normal pages, they are things like http://www.cooke.co.uk/?post_type=crown_enquiry&p=1045 I have used the redirection tool in WordPress to redirect all these links to the homepage but does anyone know why this is happening since it is the second time I had to do it. Thanks.
Reporting & Analytics | | AAttias0 -
Bing Webmaster Tools data discrepency on traffic
Hi, For a given time period in Bing Webmaster Tools it says that combined clicks from Bing and Yahoo organic search are about 4,700 total, for the same time period in Google Analytics, combining sessions for Bing and Yahoo Organic comes out to about 8,900 total. Has anyone else experienced this discrepancy? Is this common? How can I get 8900 visits from 4700 clicks?
Reporting & Analytics | | IrvCo_Interactive0 -
Get a list of robots.txt blocked URL and tell Google to crawl and index it.
Some of my key pages got blocked by robots.txt file and I have made required changes in robots.txt file but how can I get the blocked URL's list. My webmaster page Health>blocked URL's shows only number not the blocked URL's.My first question is from where can I fetch these blocked URL's and how can I get them back in searches, One other interesting point I see is that blocked pages are still showing up in searches.Title is appearing fine but Description shows blocked by robots.txt file. I need urgent recommendation as I do not want to see drop in my traffic any more.
Reporting & Analytics | | csfarnsworth0 -
Google Analytics - New account creation broken
Is it just me or is the new Google Account Creation button not working. After i set it up, it just simply doesn't save.
Reporting & Analytics | | TheGrid0 -
Google Analytics - my continuing adventures
Hello I'd appreciate views of the various metrics I'm struggling with in GA: I've run 2 different reports that provide 2 different outputs. 1. In Standard Reporting you can report in Traffic Sources on Organic Search by Keyword, which returns the number of Visits. 2. In Custom Reporting you can define the Keyword dimension and the Organic Searches metric, which returns the number of Organic Searches. This returns 2 different numbers. For example, over the last month for a given term report 1 returns 77,306 visits whilst report 2 returns 52,589 organic searches. I have found some definitions: "Visits represent the number of individual sessions initiated by all the visitors to your site." "Organic Searches: number of organic searches that happened within a session. This metric is search engine agnostic." My understanding of these definitions is that report 2 should return a larger value than report 1 rather than what is happening (i.e. report 1 returns a greater value than report 2). Does anyone have a greater understanding of what these mean and relate to? Does anyone have any views on which metric is more useful? Thanks Neil
Reporting & Analytics | | mccormackmorrison0 -
Google Analytics
Hello In Google analytics you can obtain the number of visits as a result of non-paid search. You can also set up custom reports to find the number of organic searches. The numbers are different, so what is the obvious difference between these two metrics that I'm missing. Thanks in advance for any assistance. Neil
Reporting & Analytics | | mccormackmorrison0