Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
In google analytic for google /cpc it is showing url with 404 which even not exists in my database
Hello All, In google analytic for google /cpc it is showing url with 404 which even not exists in my database that also more than 300 per day. How can it is possible? it is showing /black-friday-offers but I don't have such page. Thanks!
Reporting & Analytics | | pragnesh96390 -
Most of the goal conversion in Google analytics showing under referral
Hi All, I have already asked this question twice here but issue is not resolved yet. http://moz.com/community/q/all-goal-conversion-in-google-analytics-showing-under-referral http://moz.com/community/q/some-goal-conversion-in-google-analytics-showing-under-referral In Google analytics most of the goal conversion in Google analytics showing under referral from payment gateway sites (Paypal.com & epdq.co.uk). I have applied all the possible solution that suggested by Moz community experts & Google Adwords support team but it is not resolved yet and it is really weird answer to give client that I have applied all possible solution and issue is still not solved. Lastly I am sharing Cross domain tracking code here that I have used on my site. Please check once and let me know if code is correct and If not correct please help me out to correct the code or suggest any other way to fix the issue Cross domain tracking code ======================= <script type="text/javascript">(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
Reporting & Analytics | | Alick300
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('require', 'linker');
ga('linker:autoLink', ['paypal.com', 'payments.epdq.co.uk'], false, true);
ga('create', 'UA-xxxxxx', 'auto', { 'allowLinker': true});
ga('require', 'displayfeatures');
ga('send', 'pageview');</script > I have hide the UID in above code. Thanks0 -
No option to connect to google analytics!
I figured it out had to click on the pencil icon. On GA looks like this http://orlandocouponsfree.com - http://orlandocouponsfree.comUA-9313894-1orlandocouponsfree.combut url is http://www.orlandocouponsfree.com/Is this an issue? When I tried to connect to google analytics yesterday. There was a connection issue with Moz & GA. I refreshed page hoping it would help. Now it won't let me choose domain. When I view campaign it says I should connect properties. [IMG]http://i.imgur.com/jkJ0ZdF.png[/IMG] I click link and get to this page which doesn't provide my an option. http://analytics.moz.com/settings/campaign/89561.117561 [IMG]http://i.imgur.com/jkJ0ZdF.png[/IMG]
Reporting & Analytics | | touristips0 -
Roger says I have 378 404 errors
Hi The report shows that I have 404 errors in the about of almost 300 but they appear to me to be active and part of the site. ??? Examples: NOTE: warning - these pages discuss BDSM and have some nude images here are some on the list http://bestslavetraining.com/groups/discipline-and-punishment/?acpage=6 http://bestslavetraining.com/groups/general-b-d-s-m/?acpage=13 also here is one that does not work http://bestslavetraining.com/activity/activity/ WHY does it show 404 errors for urls that work??? what am I missing and what in the world am I to correct if they appear on the SEOmoz error log.???
Reporting & Analytics | | Cuffsmaster0 -
Google Analytics: Deleted Profile
Has anyone ever successfully managed to have a deleted GA profile restored? One of our client's profiles was deleted accidentally. I know the official line is it can't be restored, but...
Reporting & Analytics | | David_ODonnell0 -
Duicated page error
Hi, I am trying to figure out how to fix duplicated error Most of them are from wordpress "feed" Does anyone know how to fix this problem? | Wedding Photographer San Antonio | Soobumim Photography 210-863-9878 begin_of_the_skype_highlighting 210-863-9878 end_of_the_skype_highlighting http://www.soobumimphotography.com/feed/?paged=11 21 1 0 Wedding Photographer San Antonio | Soobumim Photography 210-863-9878 begin_of_the_skype_highlighting 210-863-9878 end_of_the_skype_highlighting http://www.soobumimphotography.com/feed/?paged=12 21 1 0 Wedding Photographer San Antonio | Soobumim Photography 210-863-9878 begin_of_the_skype_highlighting 210-863-9878 end_of_the_skype_highlighting http://www.soobumimphotography.com/feed/?paged=13 |
Reporting & Analytics | | BistosAmerica0 -
Editing Google Analytics eCommerce Javascript for Volusion
Does anyone know how to edit the ecommerce javascript with Volusion dynamic variables? I thought I did it correctly but none of the information got recorded in GA. It recorded that there was a transaction made but not info on the purchase price, order #, etc. This is the current code I am using: NOTE: In Volusions customer support center they provided this information: Where Order-Specific Information is Generated Order-specific information can be generated by JavaScript on the OrderFinished.asp page within Volusion. On this page, two JavaScript arrays will be initialized that advanced users may need to be aware of. The arrays and their contents are as follow:
Reporting & Analytics | | jhooley
Order Array Contents Order[0] = Order ID
Order[1] = Unused
Order[2] = Payment Amount
Order[3] = Affiliate Commisionable Value
Order[4] = Sales Tax
Order[5] = Total Shipping Cost
Order[6] = Billing City
Order[7] = Billing State
Order[8] = Billing Country OrderDetails Array Contents OrderDetails[X][0] = Order ID
OrderDetails[X][1] = Order Detail ID
OrderDetails[X][2] = Product Code
OrderDetails[X][3] = Product Name
OrderDetails[X][4] = (unused)
OrderDetails[X][5] = Product Price
OrderDetails[X][6] = Quantity Note that in the above example, X refers to a line item number (beginning from zero). Note that there can be multiple Order Details arrays - one for each item contained within an order. I'm a newbie at Analytics installation and could really use any help. Thanks!1 -
Organic search on google
Hi there, pl take a look at this link, there is a section which says shared results and has a star against two agencies and once article. can pl someone let me know what those starts are and how to get them? Thank you 🙂 so sorry - the link is http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=digital+mareketing+agency+los+angeles Vijay
Reporting & Analytics | | vijayvasu0