Our crawler was not able to access the robots.txt file on your site.
-
Good morning,
Yesterday, Moz gave me an error that is wasn't able to find our robots.txt file. However, this is a new occurrence, we've used Moz and its crawling ability many times prior; not sure why the error is happening now. I validated that the redirects and our robots page are operational and nothing is disallowing Roger in our robots.txt.
Any advice or guidance would be much appreciated.
https://www.agrisupply.com/robots.txt
Thank you for your time.
-Danny
-
Hi,
Aren't you just allowing Googlebot to scan /images folder?
-
Try changing the user agent to googlebot and see if it works
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Magento Site "Title Missing or Empty"
Hi everyone . . ..bear with me as I am "noob". My moz analysis brought up a few "critical" issues, one of which was a missing or empty title for a page. That page is: http://www.thirdcoastsigns.com/sales/guest/form Which does not even appear to be a working page . . . so I'm at a bit of a loss ow to fix this. I suspect it's a little bit of SEO expertise and a little bit of Magento expertise. Thoughts and suggestions welcome! P.S. Look forward to my NEXT noob question about a bunch of pages I have flagged as "Pages with temporary redirects". Stay tuned!
Moz Pro | | damon12120 -
Is it still best practice to optimize your site with geographic long tail keywords?
Since Google is tailoring search results to user IP and location, is it still best practice to optimize your site titles etc. with long-tail geographic keywords? For example, instead of optimizing a page for "dentist in West Palm Beach, Florida", search users who are IN West Palm Beach can just search "dentists" and a list of local dentists will be displayed (both in the local listings AND organic search listings). I'd love to see Rand cover this on a Whiteboard Friday!
Moz Pro | | RickyShockley1 -
Find out what keywords a site is ranking for
I want to find out what keywords a site is ranking for. In other words, I want to find out how Google is currently seeing a site. I've used Google KW tool - not very accurate when you simply type in the URL. Is there anything in SEOMoz I can use? New to this. thanks
Moz Pro | | goodge0 -
What user agent is used by SEOMOZ crawler?
We have a pretty tight robots.txt file in place to only allow the major search engines. I do not want to block SEOMOZ.ORG from being able to crawl the site so I want to make sure the user agent is open.
Moz Pro | | eseider0 -
Is my Site Spam?
Recently google dropped our site a big time. Can some body tell me if my site is spammy. Our visibility was 67% and one of our top competitor had the visibility of 72%. www.aa-rental.com
Moz Pro | | tanveer10 -
How to find the authorities site to link from
Hi It seems getting quality authority backlinks is the main priority to success in your SEO which is also most difficult task as well . How do you find these authority sites that allows you to backlink from them? is there any tools to find good quality sites for backlinks? Thanks
Moz Pro | | Matt-kind0 -
So many problems with my site.
Hi all.I was shocked when when i run a campaign and the warnings and recommendations about my site are so many.I know nothing about web design and the person who design it is asking me what are these problems and where did i get all these? any solution this are the problems 1.5XX (Server Error)
Moz Pro | | jubba
2.Duplicate Page Content(875)
3.Duplicate Page Title(875)
4.Overly-Dynamic URL(1048)
5.Too Many On-Page Links(60) and this is just a few of the problems.0 -
Is there such thing as a site free from errors?
Or is this a given? I am new to SEO and SEOmoz. One of my campaigns is completley free of errors...the others are a work in progress. Now I realize that SEO is never done, but can a site actually be free of errors? If so... I just gave myself a pat on the back.
Moz Pro | | AtoZion0