Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Our crawler was not able to access the robots.txt file on your site.

Moz Pro

659

Danny_Gallagher last edited by

Good morning,

Yesterday, Moz gave me an error that is wasn't able to find our robots.txt file. However, this is a new occurrence, we've used Moz and its crawling ability many times prior; not sure why the error is happening now. I validated that the redirects and our robots page are operational and nothing is disallowing Roger in our robots.txt.

Any advice or guidance would be much appreciated.

https://www.agrisupply.com/robots.txt

Thank you for your time.

-Danny
1 Reply Last reply
Reply Quote 0
Optimal_Strategies last edited by

Hi,

Aren't you just allowing Googlebot to scan /images folder?
1 Reply Last reply
Reply Quote 0
Jason-Rogers last edited by

Try changing the user agent to googlebot and see if it works
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Is your live site supposed to have rel canonical tags?

I recently started working for a company and got them to use Moz and I have found that our secure site and our live sites are creating "duplicate content" according to the Crawl Diagnostics feature. On our secure site we have rel canonical tags pointing to our live site. I'm not super familiar with rel canonical tags, but our developer says we're doing the right thing. Would love any insight you guys may have if this is actually duplicate content or not. Thanks so much!
Moz Pro | | Chase_Cleckner

0
Compare sites?

I'm frustrated, so want to ask a stupid question....My site.. www.seadwellers.com outranks my biggest competitor in most Moz catagories... www.rainbowreef.us ...EXCEPT Facebook likes...(he has a ton) **And yet, rainbowreef.us outranks me in most keywords on
Google?! I know it's not simple...but Can anyone take a quick peek and give me any insight as to why??? ** Example "Dive Key Largo" keyword...he is #1 and I am #5...typical in the most important keywords!
Moz Pro | | sdwellers

0
Broken CSV files?

Hi, I have been downloading fiels from seoMOZ for quite a while and I keep getting broken CSV files. ON my Mac, I wrote a script that cleaned them out but I cannot (don't know how) do this on the Windows side. The problem is that some of the record lines get broken up mainly in the "Anchor Text" field (maybe the anchor text in question contained a return). This of course messes up my file when I open it in Excel. Is there a way get/open these files without having them messed up like this? P..S. I am also seeing problem with text encoding, I would like to see the anchor text that is being diaplayed on the referring site rather than its HTML/Unicode/UTF-8 code value. TIA, Michel
Moz Pro | | analysestc

0
I am looking for SEO tips specifically for magazine site's

I have a client who has a website that is based on a magazine. They make their money through advertisement I am primarily an inbound marketer I would be very grateful if anyone out there has any tips for a site that has been around for quite a while ( over 10 years) we are transforming the site from HTML into WordPress then hosting it with a fast managed WordPress host using CDN. I feel the lack of links is an obvious place to start however if there's anything specific to magazine based websites I would be more than grateful to hear your opinions. Thank you all in advance. Sincerely, Thomas von Zickell
Moz Pro | | BlueprintMarketing

0
SEO Web Crawler - Referrer Lists XML Sitemap URL

Hello!, I recently ran the crawl tool on a client site. Opening up the file, I noticed that the referring URLs listed are my XML sitemaps and not (X)HTML pages. Any reason or thoughts behind why this is happening? Thanks!
Moz Pro | | MorpheusMedia

0
How do i get rid of a duplicate page error when you can not access that page?

How do i get rid of a duplicate page error when you can not access that page? I am using yahoo store manager. And i do not know code. The only way i can get to this page is by copying the link that the error message gives me. This is the duplicate that i can not find in order to delete. http://outdoortrailcams.com/busebo.html
Moz Pro | | tom14cat14

0
How long does it take Open Site Explorer to register my backlinks?

I have been using Open Site Explorer for the following website http://www.doorhandlesltd.co.uk/ I have been link building for serveral months now and added over 100 links including directories such as business.com yet when I run the Open Site Explorer report it says I only have 10 root domain links.......the ones that were there before I started my link building. Any ideas on why this might be? Or does Open Site Explorer take a certain amount of time to collect this data? If so does anybody know how long this takes? Thanks
Moz Pro | | BelfastSEO

0
How to set up SEOMOZ to track multilanguage sites?

Hi, I am managing a site in 5 different languages/regions. Our language structure is as follows: www.domain.com/en-us/ - For US www.domain.com/en-uk/ - For UK www.domain.com/fr/ - For France ... However, SEOMOZ does not allow to set up a domain with "/" as a campaign. How can I do it? Thanks
Moz Pro | | hockerty

1