Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Our crawler was not able to access the robots.txt file on your site.

Moz Pro

659

Danny_Gallagher last edited by

Good morning,

Yesterday, Moz gave me an error that is wasn't able to find our robots.txt file. However, this is a new occurrence, we've used Moz and its crawling ability many times prior; not sure why the error is happening now. I validated that the redirects and our robots page are operational and nothing is disallowing Roger in our robots.txt.

Any advice or guidance would be much appreciated.

https://www.agrisupply.com/robots.txt

Thank you for your time.

-Danny
1 Reply Last reply
Reply Quote 0
Optimal_Strategies last edited by

Hi,

Aren't you just allowing Googlebot to scan /images folder?
1 Reply Last reply
Reply Quote 0
Jason-Rogers last edited by

Try changing the user agent to googlebot and see if it works
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Moz can't see my link from an external site

Hi, I'm hoping someone can shed some light on this. I've had a followed link created from http://www.dubaisharks.com/ pointing to my website - http://www.yallarugby.com but in the reports it doesn't show that there is a link coming fro this website. The link has been active for a few months now, but still no record of it. Any ideas?
Moz Pro | | SeoSheikh

0
How do I find out which pages are being indexed on my site and which are not?

Hi, I doing my first technical audit on my site. I am learning how to do an audit as i go and am a lost. I know some page won't be indexed but how do I: 1. Check the site for all pages, both indexed and not indexed 2. Run a report to show indexed pages only (i am presuming i can do this via screaming Frog or webmaster tool) 3. I can do a comparison between the two list and work out which pages are not being indexed. I'll then need to figure out way. I'll cross this bridge once i get to it Thanks Ben
Moz Pro | | benjmoz

0
Having 1 page crawl error on 2 sites

Help! A few weeks back, my dev team did some "changes" (that I don't know anything about), but ever since then, my Moz crawl has only shown one page for either http://betamerica.com or http://fanex.com. Moz service was helpful in talking about a redirect loop that existed, and I asked my team to fix it, which it looks to me like they have. Still, 1 page. I used SEO Book's spider tool and it also only sees 1 page, and sees the sites as http://https://betamerica.com (for example), which is just weird. I don't know enough about HT Access or server stuff to figure out what's going on, so if someone can help me figure that out, I'd appreciate it.
Moz Pro | | BetAmerica

0
I am a PRO member. But will not be able to use the PRO feature. Why?

I am a PRO member. But will not be able to use the PRO feature. Why?
Moz Pro | | roadmap

0
Why is my site not ranking?

I have done various link building strategies, and used all white hat seo methods. SEOMoz even shows that my site is better than all my competition, but yet it doesn't rank ANYWHERE on Google. Can someone point me in the right direction as to why this is happening? I have attached the opensitexplorer screenshot showing my competition. 2013-02-13_1456_zpsb86ab0e1.png
Moz Pro | | locallyrank

0
File permissions save. any suggestions

Hi - something not related directly to SEO - but a vital factor Need to know for any tool - by which i can take backup or save existing file permissions of all my web site files.. God forbid - in event of any unfortunate circumstance if some mis hap happens or file permissions changed accidentally - how can i get them back to the original. Any tool by which i can save existing file permissions of filezilla thanks
Moz Pro | | Modi

0
Open site explorer result discrepancy

Hi all, I'm a little confused - when using open sites explorer to view of link stats on one of our competitors site I get conflicting results. For example - Page Specific Metircs shows the url www.xyz.com has 265 external links. But Root Domain Metrics show they have 22,687 links Yet the total links at the top of the report says Total links 999 Can someone explain why www.xyz.com has 265 external links and the rot domain (which is the same) has 22687 links? I've also run the report on xyz.com rather than www.xyz.com which yielded no results. Thanks in advance
Moz Pro | | EclipseLegal

0
Confounding "Accessible to Engines" error?

Most of the pages on our site "Accessible to Engines" test in the SEOmoz reports. We cannot find any problem with the code and it's largely identical to the few pages that come up with an "A" score. One item that may be a reason is that we use meta http-equiv="refresh" content="600; For example in www.weatherzone.com.au/nsw/sydney/sydney We use this to fresh dynamic content on our site. Do search engines penalise pages that use this form of page refresh? Alternatively, is there a known bug in the SEOmoz "Accessible to Engines" report? Many thanks
Moz Pro | | weatherzone

0