Our crawler was not able to access the robots.txt file on your site.
-
Good morning,
Yesterday, Moz gave me an error that is wasn't able to find our robots.txt file. However, this is a new occurrence, we've used Moz and its crawling ability many times prior; not sure why the error is happening now. I validated that the redirects and our robots page are operational and nothing is disallowing Roger in our robots.txt.
Any advice or guidance would be much appreciated.
https://www.agrisupply.com/robots.txt
Thank you for your time.
-Danny
-
Hi,
Aren't you just allowing Googlebot to scan /images folder?
-
Try changing the user agent to googlebot and see if it works
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my site not crawl?
hi all me allow rogerbot in robots.txt but rogerbot can't crawl my site my site: toska-co.ir
Moz Pro | | jahanidawodi0 -
Size of a term to make money off of ad based informational site
Hello, I am used to estimating how much money is possible to make out of ranking for Ecommerce terms. Higher price products have a lower conversion rate (cheap items are opposite) so you can look at the keyword size in keyword explorer and estimate pretty well (roughly). My question is, how many times bigger does an ad based (informational) keyword need to be to get similar profit? This is just an estimate. For example, does a term need to be 5? 10? times bigger on average (volume) as an informational keyword to make the same amount of profit (roughly of course)? I realize there are many factors here but you must have some way you think of it? Can you give me some general guidelines?
Moz Pro | | BobGW0 -
SEOmoz crawler not crawling my site
We set up a new campaign in SEOmoz on Friday. It is my understanding that the preliminary crawl can cover up to 250 and this has been our experience in the past. However, the preliminary crawl only went through 2 pages. This is a larger eCommerce site with many pages. Any ideas why more pages weren't crawled? We set up the campaign to track at the root domain level.
Moz Pro | | IMM0 -
Open Site Explorer Twitter Stats
Hi, I have just noticed something within Open Site Explorer. On the page it says it draws the Twitter stats from tweetmeme. I have just noticed that tweetmeme have shut down. Their new product datasift seems to do the same thing, but can someone confirm if SEOMoz is drawing data from them now, or is the current twitter data shown not fully up to date. Its not a major issue, its just something I noticed. Thanks Rich
Moz Pro | | mos_rich0 -
Broken CSV files?
Hi, I have been downloading fiels from seoMOZ for quite a while and I keep getting broken CSV files. ON my Mac, I wrote a script that cleaned them out but I cannot (don't know how) do this on the Windows side. The problem is that some of the record lines get broken up mainly in the "Anchor Text" field (maybe the anchor text in question contained a return). This of course messes up my file when I open it in Excel. Is there a way get/open these files without having them messed up like this? P..S. I am also seeing problem with text encoding, I would like to see the anchor text that is being diaplayed on the referring site rather than its HTML/Unicode/UTF-8 code value. TIA, Michel
Moz Pro | | analysestc0 -
How can I download all the inbound links to my Site?
Hi, From Site Explorer, I want to download all the inbound links linking to my site: www.comm100.com/livechat/. But it seems that there is a limitation. I can only download a part of the inbound links? What should I do? Or is it because my account is free trial? Do I have to pay for it? Please help. Thanks,
Moz Pro | | Sophia_M0 -
Is anybody else having problems with Open Site Explorer downloads
I've been trying to get a CSV for linking domains for two days now and still have nothing. It got to the "Finalizing" status within a few hours and stayed that way for about 24 hours, and now it's back to the beginning of a "pending" status.
Moz Pro | | BlastAM0 -
Can i force another crawl on my site to see if it recognizes my changes?
i had a problem w/dup content and titles on my site, i fixed them immediately and im wondering if i can run another crawl on my site to see if my changes were recognized thanks shaun
Moz Pro | | daugherty0