Still Cant Crawl My Site

[martJ]

I've removed all blocks but two from our htaccess. They are for amazonaws.com to block amazon from crawling us.

I did a fetch as google in our WM tools on our robots txt with success.

SEOMoz crawler here hit's our site and gets a 403. I've looks in our blocked request logs and amazon is the only one in there.

What is going on here?

[ShaMenz]

Hey Joel,

Happy Friday!

Sha

[ShaMenz]

Hi Dana,

No problem. Glad you have sorted the problem now.

Have an awesome weekend

Sha

[JoelDay]

Hey Dana,

We've been corresponding in email, but I just wanted to update your thread here as well.

We don't use Amazon's bot, we use Amazon Web Service to host our crawler. If you are no longer blocking AWS you should be able to crawl OK moving forward.

Thanks!
Joel.

[martJ]

Wish someone would've pointed that out days ago.

Thank you soooooo much for your great answer.

I don't understand though how or why seomoz is using amazons bot...

What if I don't want amazon accessing our site ( i dont). That means we can't use seomoz then??

[martJ]

we'll see how this goes. I've removed the blocks for amazonaws...

Thanks .

[ShaMenz]

Hi Dana,

I believe SEOmoz utilizes Amazonaws services for crawling, (or at least they did a few months ago) so that may well be your problem.

The best (and quickest) way to confirm this is to go to the SEOmoz Help Hub and click the button at the top of the page to contact the Help Team directly.

Hope that helps,

Sha

[iQandil]

Whats the web address?

Issa