Our crawler was not able to access the robots.txt file on your site.
-
Good morning,
Yesterday, Moz gave me an error that is wasn't able to find our robots.txt file. However, this is a new occurrence, we've used Moz and its crawling ability many times prior; not sure why the error is happening now. I validated that the redirects and our robots page are operational and nothing is disallowing Roger in our robots.txt.
Any advice or guidance would be much appreciated.
https://www.agrisupply.com/robots.txt
Thank you for your time.
-Danny
-
Hi,
Aren't you just allowing Googlebot to scan /images folder?
-
Try changing the user agent to googlebot and see if it works
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved How many sites can I track with one subscription?
Hello, We are currently a MozPro medium member and we are tracking amlrightsource.com but we have other sites we'd like to track as well. Wondering if we can track more sites with this subscription?
Moz Pro | | KassandraSharr0 -
How to sift "site search" data from Google Analytics for trends
I apologize in advance if this has been asked a million times but I'm just not able to find anything on it for some reason. Probably the words "site" and "search" come up a lot in this area... Anyhow, my question: How do I find trends in "site search" data from Google Analytics? I set up "site search" a long time ago. I have thousands and thousands of searches people have made on my site logged and squirreled away. The plan was to review them on a weekly basis, find the trends and start writing content to address interests people seem to be having but not finding on our site. Sounded great at the time. The problem I have, of course, is that among my 10,000 searches (many shown in Google Analytics as "no-results:cats and dogs", etc), there are slight differences that make it difficult to total up search trends. Let's say the list is like this: Term | Search Count Cats | 500
Moz Pro | | rtkl
Dogs | 500
Cat | 250
Dog | 250
Cat food | 5
Dog food | 5
Birds | 1
Bird | 1
Cats are great | 1
Cats are really great | 1
Dogs are great | 1
I like birds | 1
Seriously, I like Cats | 1
Turtles | 1 ... 10,000 more entries, every single one only 1 search per term. OK, so it looks like people like Cats and Dogs a lot, but also Birds and Turtles. But maybe there are snake searches. Maybe there are "cat pajamas" searches and variations on all of the above. Who knows what else is really trending in there??? The review of this data is MIND-NUMBING. Especially when you get into plurality and misspellings, this rabbit hole has no bottom. Is there a tool people in the SEO jam use to take a big ole CSV dump and have it magically sorted by at least potential trends? I mean, there's gotta be, right? And I'm silly for not already knowing what it is.0 -
How to create effective Backlinks and promote very small sites?
I want to create effective backlinks and promote websites with having only 8-15 pages with very poor/basic content. Where client is unable to provide content, increase number of pages etc. So kindly suggest..
Moz Pro | | 1akal0 -
Where is the best place to add links on my site?
If I'd like to put links to other sites in my site, is it better to have a page named "Our Helpful Links" etc. instead of just adding them to the bottom of an existing page like I've seen on some sites? I'm asking because I'm wanting to make Google as happy as possible and still add them. Just in case it helps to look at the site yourself to give advise its; http://www.allstatetransmission.net If you see anything else there that I should work on feel free to be hard on it, I value any criticism. Thanks, Jeff
Moz Pro | | allstatetransmission0 -
How to force a recrawl of a site?
Hi, I made changes in my site. I would like to see the result of the crawl diagnostic. I know the crawl is happening every week, however, is there a way to force a re-crawl in order not to have to wait 5 days? Cheers,
Moz Pro | | nuxeo0 -
How to get seomoz to re-crawl a site?
I had a lot of duplicate content issues and have fixed all the other warnings. I want to check the site again.
Moz Pro | | adamzski0