Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Our crawler was not able to access the robots.txt file on your site.

Moz Pro

661

Danny_Gallagher last edited by

Good morning,

Yesterday, Moz gave me an error that is wasn't able to find our robots.txt file. However, this is a new occurrence, we've used Moz and its crawling ability many times prior; not sure why the error is happening now. I validated that the redirects and our robots page are operational and nothing is disallowing Roger in our robots.txt.

Any advice or guidance would be much appreciated.

https://www.agrisupply.com/robots.txt

Thank you for your time.

-Danny
1 Reply Last reply
Reply Quote 0
Optimal_Strategies last edited by

Hi,

Aren't you just allowing Googlebot to scan /images folder?
1 Reply Last reply
Reply Quote 0
Jason-Rogers last edited by

Try changing the user agent to googlebot and see if it works
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

How to remove broken links from our wordpress site?

Hello! How are you? We just signed up to Moz.com. Moz link tool. It gave us many broken links with 404's and 302's. Could you please help me with deleting the links? Thanks!
Moz Pro | | hsma

0
I need an interlinking report for my site, is there a report in Moz or another application that tell me how all of my pages are linked to other pages on my site?

I am in the process of doing a redesign for one of my sites. I need an interlinking report for my site. Is there a report in Moz or another application that tell me how all of my pages are linked to other pages on my site?
Moz Pro | | seoflorida

0
We have a Wix site but I wonder if we would be better suited with square space

Our SEO has taken a hit in moving our website from homestead website builder to Wix this last year. 1,500 unique visitors a month to 400 on our website www.bestlifeint.com using much of the same SEO and content. I think 2 of our main issues now (site speed and SEO friendliness) would be addressed in the switch to squarespace. But I guess is it worth it? I do not think google likes the whole hash bang html5 ajax thing wix uses and our website relies heavily on good seo. Any advice? I am starting link building but it might be a waste if we switched the site and in turn changed all the page url's as wix uses ?! for it's pages. Just need some clarification ideas on getting traffic to our website, also SEO MOZ software is hard to use report card with wix. We also switched our ecommerce from homestead to shopify and this has been a good move. Shopify (our ecommerce actually gets more traffic than our website now. But the problem I feel is wix hurts our seo just by being on their platform. Seriously thinking of squarespace but feel we would have to start from scratch and just match some of the elements on current site. Any insight, ideas? Am I on the right path in my thinking? Should I just stick it out with Wix? Thanks. -Brian
Moz Pro | | SammisBest

0
Robots.txt

I have a page used for a reference that lists 150 links to blog articles. I use in in a training area of my website. I now get warnings from moz that it has too many links. I decided to disallow this page in robots.text. Below is the what appears in the file. Robots.txt file for http://www.boxtheorygold.com User-agent: * Disallow: /blog-links/ My understanding is that this simply has google bypass the page and not crawl it. However, in Webmaster Tools, I used the Fetch tool to check out a couple of my blog articles. One returned an expected result. The other returned a result of "access denied" due to robots.text. Both blog article links are listed on the /blog/links/ reference page. Question: Why does google refuse to crawl the one article (using the Fetch tool) when it is not referenced at all in the robots.text file. Why is access denied? Should I have used a noindex on this page instead of robots.txt? I am fearful that robots.text may be blocking many of my blog articles. Please advise. Thanks,
Ron
Moz Pro | | Rong

0
How to find the authorities site to link from

Hi It seems getting quality authority backlinks is the main priority to success in your SEO which is also most difficult task as well . How do you find these authority sites that allows you to backlink from them? is there any tools to find good quality sites for backlinks? Thanks
Moz Pro | | Matt-kind

0
Moving from a dynamic site to nopcommerce

Hi, First question since becoming a member so be gentle with me ;o) We are moving from a site using a dynamically generated ecommercetemplate to a nopcommerce site. I have two questions about this I know that 301 redirects are the best way to pass "link juice" but with the site being dynamically generated a lot of these links will simply disappear when we move to nop, meaning that they wont actually be there in order to add a 301. Any advice on this would be appreciated. How can i get a list of the pages on my current site which have the best rank in order to create the redirects. Unfortunately due to some technical issues we had with google analytics we were unable to install it so dont really have any analytics to give us some extra info. I was hoping that there would be somewhere within seomoz where i could be directed. Many thanks Chris
Moz Pro | | cjhamill

0
No Local Directory links in Open Site Explorer?

Hi there, I just signed up for seomoz pro today and have been playing around in OSE looking at a number of local business websites that I manage SEO for. For all of them, I have put an endless amount of effort ensuring they have good quality, accurate business listings in all the local directories like yelp, yellowpages, hotfrog, tripadvisor, etc. For most of them, I did this months or even up to a year ago. However, I don't see a single link reported in OSE that is a result of my efforts here. Does OSE not report links from local business directories, or should I be concerned that I'm somehow not optimizing properly, or are these types of links of such little value that they don't get picked up, something else? Thanks for the help!
Moz Pro | | rianDS

1
Anyone else have trouble with Open Site Explorer

Ever since the new Open Site Explorer came out, I've haven't been able to use the download feature. It just gives me a prompt saying "No Data Received." Anyone else have trouble with OSE, or been able to get past this issue?
Moz Pro | | EricVallee34

1