Still Cant Crawl My Site
-
I've removed all blocks but two from our htaccess. They are for amazonaws.com to block amazon from crawling us.
I did a fetch as google in our WM tools on our robots txt with success.
SEOMoz crawler here hit's our site and gets a 403. I've looks in our blocked request logs and amazon is the only one in there.
What is going on here?
-
Hey Joel,
Happy Friday!
Sha
-
Hi Dana,
No problem. Glad you have sorted the problem now.
Have an awesome weekend
Sha
-
Hey Dana,
We've been corresponding in email, but I just wanted to update your thread here as well.
We don't use Amazon's bot, we use Amazon Web Service to host our crawler. If you are no longer blocking AWS you should be able to crawl OK moving forward.
Thanks!
Joel. -
Wish someone would've pointed that out days ago.
Thank you soooooo much for your great answer.
I don't understand though how or why seomoz is using amazons bot...
What if I don't want amazon accessing our site ( i dont). That means we can't use seomoz then??
-
we'll see how this goes. I've removed the blocks for amazonaws...
Thanks .
-
Hi Dana,
I believe SEOmoz utilizes Amazonaws services for crawling, (or at least they did a few months ago) so that may well be your problem.
The best (and quickest) way to confirm this is to go to the SEOmoz Help Hub and click the button at the top of the page to contact the Help Team directly.
Hope that helps,
Sha
-
Whats the web address?
Issa
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How Can I Solve a Spam Score Of 33% On A Site
Goof Day pls Am having S Spam Score Of 33% On my Blog Naijazeal.Com And I Check In my Google webmaster and i analyze all the link i see and i see little with high spam Score But Checking My Spam Score on Moz, It showi Extermal link below but most of the link showing below is not in my Seach console Pls help
Moz Pro | | DAMIDEZ0 -
Is it still best practice to optimize your site with geographic long tail keywords?
Since Google is tailoring search results to user IP and location, is it still best practice to optimize your site titles etc. with long-tail geographic keywords? For example, instead of optimizing a page for "dentist in West Palm Beach, Florida", search users who are IN West Palm Beach can just search "dentists" and a list of local dentists will be displayed (both in the local listings AND organic search listings). I'd love to see Rand cover this on a Whiteboard Friday!
Moz Pro | | RickyShockley1 -
Open Site Explorer - Image Anchor Text
Hey there. I'm researching internal links and In my report I get alot of image links with the following text in the Link Anchor Text column: (img alt)(img)[No Anchor Text]. Is it because my images don't have alt-text or something else... How to optimize correctly? See attached screenshot. Best regards K87xLN8
Moz Pro | | nosuchagency0 -
Why is my site not ranking?
I have done various link building strategies, and used all white hat seo methods. SEOMoz even shows that my site is better than all my competition, but yet it doesn't rank ANYWHERE on Google. Can someone point me in the right direction as to why this is happening? I have attached the opensitexplorer screenshot showing my competition. 2013-02-13_1456_zpsb86ab0e1.png
Moz Pro | | locallyrank0 -
Order of urls in SEOMoz crawl report
Is there any rhyme or reason to the order of urls in the SEOMoz crawl report, or are the urls just listed in random order?
Moz Pro | | LynnMarie0 -
Open Site Explorer
Hi - open site explorer always showed backlinks to my site's blog for Wordpress, a couple of edu sites and others that ranked somewhat high. These are no longer being shown. However, they do show up on some other backlink check tools. Does anyone know why and if this should be a concern? Thanks. Don www.uniquegiftsanddecor.com
Moz Pro | | uniquegifts-2778790 -
What do you use for site audit
What tools do you use for conducting a site audit? I need to do an audit on a site and the seomoz web crawler and on page optimization will takes days if not a full week to return any results. In past Ive used other tools that I could run on the fly and they would return broken links, missing htags, keyword density, server information and more. Curious as to what you all use and what you may recommend to use in conjunction with the moz tools.
Moz Pro | | anthonytjm0 -
Crawl Test produced only 1 page
Hi, I recently submitted a crawl for www.cirrato.com using SEOMoz Crawl Test Tool. I have a lot of pages, but the crawl result shows only 1 page, which is the front page and nothing else... Does anyone know what this could mean or what the problem is?
Moz Pro | | yusufcirrato0