Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Query on google analytic benchmarking report
Hi All, First I select My Industry Vertical - ABCD then I select Region - US ( all region) then size by daily session - 10000-99999 finally reports appears "Blank" but when I decrease daily session from 10000-99999 to 1000-4999 or less then report is perfect. So what does it mean? My Avg daily session is 70k to 80k. So how to analysis benchmarking in this case? Thanks!
Reporting & Analytics | | pragnesh96390 -
Google Analytics goals by source report?
Hello everybody. Is there way in Google analytics to create report on what goals have been completed per each source? Example: Lets say I have 3 goals: Subscription, Purchase, Quote. How can I get report, saying something like this: google / organic - Subscription - 5 conversions
Reporting & Analytics | | DmitriiK
Purchase - 3 conversions
Quote - 10 conversions and so on. P.S. Basically, I want the reverse of standard Google Analytics goal completions report, where you can click on goal and see which sources/mediums completions came from. I'd like to do the opposite - "click" on source/medium and see which goals have been completed. Thanks0 -
Google Analytics Goal Tracking Not Working
I am trying to install goal tracking for a client to measure leads from SEO, and I'm running into problems with the Google Analytics goal tracking not working. The only thing that I can think of that is preventing goal tracking from working is that the URL confirming page request URI looks like "contact/#contact-form-4475" and perhaps the hashtag is causing problems. I wanted to ask the community if any of you have run into basic conversion tracking problems like this, where you need to track form fills on a website via a confirmation URL/destination URL goal, and what you did to solve it? I appreciate any input on how to get data on conversions either by fixing the goal in GA or using a different tracking tool. Thanks!
Reporting & Analytics | | williammarlow0 -
Question about setting up Google Webmaster on Network Solutions?
I'm trying to set myself up as a Webmaster on my company's site. We use Network Solutions. I am following Google's directions on how to do this. However, I am a bit wary. Here are the directions. Underlined is the part I am having trouble with: Log in to your account for howlatthemoon.com at www.networksolutions.com by clicking theManage Account icon. In the left navigation bar, open the nsWebAddress (Domains) menu by clicking the **+ **icon. Click Manage Domain Names. On the Domain Details page for the domain you're using, select the Designated DNS radio button (to the right of Change domain to point to) and click the Apply Changes button. If you've previously modified your advanced DNS settings, click Edit (to the right ofDomain currently points to). Under the Advanced DNS Manager heading, click Manage Advanced DNS Records. Under the Text (TXT Records) heading, click Add/Edit. In the Host field, enter @. Leave the TTL field set to the default value. In the Text field, copy and paste the following unique security token:
Reporting & Analytics | | howlusa
(security token removed for obvious reasons) Click Continue. Review your changes and click Save Changes. When you've done saving the TXT record, click the Verify button below on this page. There is already a host of @ (None). The text for it reads: v=spf1 include:_spf.google.com ~all I called Network Solutions and the guy I was speaking with told me to delete it and replace it with my Google Webmasters code. However, I think this is setting up our email. Do I just add the Webmasters in and have two hosts of @ (None)? Thanks!0 -
Robots.txt file issue.
Hi, Its my third thread here and i have created many like it on many webmaster communities.I know many pro are here so badly needs help. Robots.txt blocked 2k important URL's of my blogging site http://Muslim-academy.com/ Especially of my blog area which are bringing good number of visitors daily.My organic traffic declined from 1k daily to 350. I have removed the robots.txt file.Resubmitted existing Sitemap.Used all Fetch to index options and 50 URL submission option in Bing Webmaster Tool. What Can I do know to have these blocked URL's back in Google index? 1.Create a NEW sitemap and submit it again in Google webmaster and bing webmaster tool? 2.Bookmark,linkbuilding or share the URL's.I did a lot of bookmarking for blocked URL's. I fetch the list of blocked URLS Using BING WEBMASTER TOOLS.
Reporting & Analytics | | csfarnsworth0 -
Unexplained Crawl Diagnostic Errors & Opencart
Hi, I've been looking at the crawl diagnostics for my site and trying to fix the errors that are showing up but Seomoz is producing some strange results. It's saying pages are duplicated upto 16 times but those pages dont exist. It's adding "page=3", "page=4" to the end of the product URL but I don't see how it's finding those pages, nothing on the site(as far as I can tell) is linking to them. There is no "page=3", just the one product page. Again on the duplicate content it's saying under the "other URLs" there's URLs like "http:///product-a" but again I don't see where it's finding these URLs and obviously those URL's dont work. Those three slashes aren't a typo either. So far I've reduced the amount of errors from 2,005 to 543 but the rest of them I can't make sense of. Also, what does one do when you have two products, eg: "product-a-white" and "product-a-black" to prevent Seomoz from seeing duplicates? Canonical links wont work because there's no parent item, just those two. Google Webmaster tools doesn't seem to have a problem though. Using Opencart 1.5, if it helps. Cheers,
Reporting & Analytics | | AsOneDesign0 -
Google API Tools - Next Analytics
Have you used any API tools like Next Analytics and/or can you recommend another tool that is particularly useful for a SEO?
Reporting & Analytics | | KnutDSvendsen0 -
Why is Webmaster Tools not Linking Correctly Google Analytics?
I have a global site with .com as the US site and then /country for all other markets i.e. .com/uk .com/de etc. Each site has it's own WMT profile. Each site has it's own GA profile. Since Google added Search Engine Optimization to the GA interface, I can't track these sites separately under this new feature. It seems to me that it can only be associated with a single WMT profile at a time or rather that WMT can only be associated with a single GA profile at a time. The numbers for the UK and US site are identical, but when I try and link the UK WMT account with GA it removes it from the US profile and vice versa.
Reporting & Analytics | | Red_Mud_Rookie0