Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search and Replace filter on Google Analytics?
Hello! On our GA account for one of our clients, we'd like to add a search and replace filter to the Views section of the account. The URL is www.askergoworks.com (it redirects to askergoworks.com), and Google has flagged us to have redundant hostnames. This is why we'd like to add the filter. Would the regular expression be askergoworks.com|www.askergoworks.com ? Any help would be great - I'm not a regex expert, so I really don't how to go about this. Thanks!
Reporting & Analytics | | AGILITY0 -
Google Analytics goals by source report?
Hello everybody. Is there way in Google analytics to create report on what goals have been completed per each source? Example: Lets say I have 3 goals: Subscription, Purchase, Quote. How can I get report, saying something like this: google / organic - Subscription - 5 conversions
Reporting & Analytics | | DmitriiK
Purchase - 3 conversions
Quote - 10 conversions and so on. P.S. Basically, I want the reverse of standard Google Analytics goal completions report, where you can click on goal and see which sources/mediums completions came from. I'd like to do the opposite - "click" on source/medium and see which goals have been completed. Thanks0 -
Google Tag Assistant for Chrome
I'm using the Google Tag Assistant for Chrome, and I noticed something really weird. No matter what pages I look at, the same two GA tags show up. It's weird. You can see the tag that is "working", and then there are two repeats. For example, when I look at this page, I see the GA tag that is working and then all the remarketing tags. Then I see UA-36732895-1 repeated twice. Anyone have any idea what this is? Thanks!
Reporting & Analytics | | PGD20110 -
Webmaster tools help
Looking at some Google Webmaster tools data and im a bit stumped. Can anyone tell me if the "not selected" pages should be taken from the "total indexed pages" or are they a seperate entity? The way the data is displayed on the graph leads you to believe they should be taken from the total indexed pages, yet Googles own description of "not selected" pages says that these pages are not indexed. Confused.
Reporting & Analytics | | Silkstream0 -
Multiple Domains in Webmaster Tools
Our client has four domains all redirecting to one domain. aquatron.info, www.aquatron.info, and www.aquatronhardwatersolution.com are all set up to redirect to aquatronhardwatersolution.com. I would like to make sure that we are getting the most link juice and that they are set up correctly in Google Webmaster Tools. Any help would be great. Thanks!
Reporting & Analytics | | PolarisWebSEO0 -
Tips for migrating Google News traffic?
We are about to relaunch a site that gets a lot of Google News traffic. We are not changing domain, but the site structure is changing greatly, with the URLs of both news index pages and articles being shaken up. Obviously, we've 301-ed every page to its closest equivalent on the new site. We've also got a news sitemap. As we are not changing domain, is there anything further we need to do to help protect our Google News traffic. On a related note, does anyone have a relaible way of measuring traffic from Google News listings in universal search?
Reporting & Analytics | | Dennis-529610 -
Google analytics for your mobile sub-domain
Hi All, I have just started mobile sub-domain for my desktop site. But I don't to know how to configure analytics for mobile sub-domain. I want to track result between desktop and mobile site. Can any one help me to sort out this problem? Waiting for reply
Reporting & Analytics | | Hexpress0 -
Export all links on page with anchor text
As the question says really. Is there a tool that will export all the links on a page and anchor text related to those links into Excel? Thanks.
Reporting & Analytics | | RikkiD220