Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Ad referral
I was wondering if someone could decode the jumble of a referral - this is supposedly the referal that led to a click through to my site via a product listing ad. I am trying to figure out how www.nextag.com comes in to the picture as we do not have refurbexperts even listed there? Thanks to anyone who tries/does work it out. http://www.googleadservices.com/pagead/aclk?sa=L&ai=CGXud6DmDU_qeL5THygHpuICwCaTZwMYD_Nvvv0bEwMS50wEIBhAEIOn5-gEoBVCl7P7f-v____8BYMnu8omYpPQSoAHAhIv9A8gBB8gDG6oEJ0_QwcNc5zNun_d7S5KNcMT6uPjjH_mMDkKFFgBCQ6aKICRPJVVa7MAFBYgGAaAGJoAHqPv0ApAHAeASupqdo-ypit0m&ohost=www.google.com&cid=5GhZEzUCSC6x9n2wxOdz3-mrAfSUkvHKPN3wD5yLInnlNil_&sig=AOD64_1D1z1JPYbFP0UnUglJVOfvd25RfA&adurl=http://refurbexperts.com/product/527/HP-LaserJet-P2015-Laser-Printer-RECONDITIONED%3Futm_source%3Dproductlistingads%26utm_medium%3Dadwords%26utm_campaign%3Dadwords&ctype=5&nb=0&res_url=http%3A%2F%2Fwww.nextag.com%2Fhp-p2015-laserjet%2Fproducts-html%3Fnxtg%3D116d0a1c0504-9FFEB16DE52A7E2A&rurl=http%3A%2F%2Fwww.nextag.com%2Fgoto.jsp%3Fp%3D3652%26search%3Dhp%2520p2015%2520laserjet%26t%3Dag%253D1384181795%26crid%3D48271786%26gg_aid%3D20169721025%26gg_site%3D%26gclid%3DCjgKEAjwzIucBRDzjIz9qMOB3TASJABBIwL1LHK7GcAPS6yHGpd9Kq3wsZrcPORAWD8QCWivr4W75PD_BwE&nm=11&nx=43&ny=12&is=700x181&clkt=187
Reporting & Analytics | | henya0 -
Google Webmaster Preferred Domain Settings Help
I was recently running my SEO Moz report for my domain and saw a drastic drop in page rankings on the report. After some trouble shooting, I checked my Google Webmaster Tools and saw the "Preferred Domain for my Site has changed" the last week in which I saw the drastic drop. Under the "preferred domain" area, I have it set to www.marware.com but on most of the links to my site, it reads www.marware.com/ . We also have http://marware.com , but am not sure if this carries any relevance as the preferred domain is set for www.marware.com . Does the "/" make a difference in the "Preferred Domain" section? If so, how do I update it? Is there a way to confirm it is set up correctly? In the photo attached, what does "Linked Property" "Analytics Web Property" and "Associated Webmaster Tools Site" mean? Thanks in advance! T5tSCxm
Reporting & Analytics | | Khadaran0 -
How to link Analytics to Webmaster Tools at the profile level?
What do you do when you have a few profiles under one property in analytics (for subdomains etc) but you want to link each one of them to webmaster tools? Far as I know, you can only link them at the property level. Thanks! -Dan
Reporting & Analytics | | evolvingSEO0 -
Google Webmaster Tools not displaying all backlinks
I was asked to take a look at a site that a friend redesigned and immediately plummeted in rankings. It was a flash site where the domain redirected to a flash landing page. When the site was redesigned, they removed the flash page and put in the new site on the domain URL without changing the redirect or putting a 301 in place on the old flash landing page URL. This site didn't have many backlinks to begin with, but it did ok for some important search phrases. Now when I look in Webmaster Tools, there are only 2 backlinks showing. Meanwhile I know there are more backlinks in existence, primarily from yellowpages and citysearch. I've had them put a 301 on the old flash landing page pointing at the domain URL and they've added a canonical URL. What have I missed here?
Reporting & Analytics | | BostonWright0 -
Tracking Subdomains without access to Primary Domain
Hi, I have look after mutliple sub domains on a website and want to implement GA sub domain tracking across them. I do not have however have access to the primary domain as thats controlled by a larger organisation. I'm wondering how to set this up so it tracks correctly? Do I set the domain name in my property to the primary domain, or the main subdomain? For example I have test1.example.com (which is basically a homepage), test2.example.com, test3.example.com and test4.example.com which are various sections. As stated, I don't control the root example.com so Do I use setdomainname test1.example.com in the tracking code or setdomainname example.com. Apply filters etc and then setup profiles for each subdomain? Hope this makes sense. Thanks.
Reporting & Analytics | | breaktheseal0 -
How serious are the Duplicate page content and Tags error?
I have a travel booking website which reserves flights, cars, hotels, vacation packages and Cruises. I encounter a huge number of Duplicate Page Title and Content error. This is expected because of the nature of my website. Say if you look for flights between Washington DC and London Heathrow you will at least get 60 different options with same content and title tags. How can I go about reducing the harm if any of duplicate content and meta tags on my website? Knowing that invariably I will have multiple pages with same content and tags? Would appreciate your advice? S.H
Reporting & Analytics | | sherohass0 -
AW Stats vs Google Analytics
Hey Moz Community, I am looking to get opinions on the best practice for analytics/traffic analysis. From experience I know that AW Stats reads high and Google Analytics reads low for traffic for reason in this article http://www.smartz.com/blog/2009/01/23/analytic-confusion-%E2%80%93-awstats-vs-google-analytics/ It drives me a little nuts how far off both are for some pages. I have one article that shows 100 views (GA) and AW stats shows 5 times that number of views. Any suggestions or systems you recommend? Thanks
Reporting & Analytics | | johnshearer0 -
Track Individual Organic Orders In Google Analytics
I was wondering if there is a way to track information about the individual order in google analytics. Currently I can see all of the organic traffic, rev, transactions, etc, but I would like to be able to know what those individual order numbers are, as well as be able to place test orders to see if organic tracking is correctly working. Does anyone know of a good blog walkthrough for this, or have any suggestions? Thanks (again individual organic order data not all data from a specific search engine or keyword).
Reporting & Analytics | | Gordian0