Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search Console
To the Moz Community, Should we be considering the information that Google Search Console is telling us? It is showing a dramatic drop in our SEO and our pages are not being indexed, however it is showing differently in our Moz Analytics section. Any clarification will be greatly appreciated. Many thanks Dawn
Reporting & Analytics | | DawnQ0 -
Google Analytics Goals - Button Tracking
Does anyone know if there is a really easy way to track a button in Google Analytics yourself? It seems that most button click goal setups involve some use of tricky code and I'm wondering if there is a much easier way to do this that will allow us to simply setup and track certain button clicks as goal conversions in Analytics. Your help here is much appreciated!
Reporting & Analytics | | Gavo0 -
Google Analytics - Events impacting session data
Hi Mozzers, Has anybody ever experienced event tracking significantly impacting session data? Here's a summary of what I'm finding: When I measure sessions to a particular hostname (a subdomain) in my Google Analytics report, sessions dropped by over 100% when I removed some event tracking on that subdomain (an event was triggered when a user clicked on a particular button). When I reinstate that event tracking, sessions go back up to around where they used to be. Can anybody understand why this might be happening? Thanks!
Reporting & Analytics | | A_Q0 -
Variables in Google Tag Manager
Last week I got a great answer here on how to implement GTM and cross domain tracking, now that we have that in place I'm looking for some more advice 😉 I'm wondering that if we push variabels into the data-layer of GTM, can we actually use those variables in reports as well? Do they get recorded in Google Analytics? I'd like to use some data that we push to the data layer for segmentation purposes. Anybody an idea how to achieve this?
Reporting & Analytics | | jorisbrabants0 -
Google Webmaster Tools not displaying all backlinks
I was asked to take a look at a site that a friend redesigned and immediately plummeted in rankings. It was a flash site where the domain redirected to a flash landing page. When the site was redesigned, they removed the flash page and put in the new site on the domain URL without changing the redirect or putting a 301 in place on the old flash landing page URL. This site didn't have many backlinks to begin with, but it did ok for some important search phrases. Now when I look in Webmaster Tools, there are only 2 backlinks showing. Meanwhile I know there are more backlinks in existence, primarily from yellowpages and citysearch. I've had them put a 301 on the old flash landing page pointing at the domain URL and they've added a canonical URL. What have I missed here?
Reporting & Analytics | | BostonWright0 -
How to track what people type on my text boxes on Google Analytics?
Hi there! In our website, we have a few text boxes that users need to use to complete the goal. The boxes aren't search boxes, but it's still important to us to track what people type on it. I'm looking for a way to track the data through the "event" feature in Google Analytics, but it seems that this tracker can only calculate clicks, or video views etc. Does anyone knows how to track do it?
Reporting & Analytics | | ivan.precisodisso0 -
Set up Google Analytics by product category
Does anyone know if it's possible to set up Google Analytics data by product category? We sell roughly twenty product categories on our ecommerce site. We can look at our analytics performance for the site overall and drill down to specific pages, but what we really want to see is our performance by product category. Our product categories can include up to 5000 products. Any pointers on how to do this?
Reporting & Analytics | | ironpac0 -
Does prevent links from being included in Google Webmaster linking sites report?
My client has clean links in edit from nytimes.com. The links do not have nofollow tags. Google Webmaster stopped including links from nytimes.com in the external linking domains report and we don't know why since the URL is still live. The nytimes.com URL includes this tag in the source code: Are links on pages with NOARCHIVE still counted in Google Webmaster linking domains reports?
Reporting & Analytics | | ebenthurston0