Welcome to the Q&A Forum

Wish someone would've pointed that out days ago.

Thank you soooooo much for your great answer.

I don't understand though how or why seomoz is using amazons bot...

What if I don't want amazon accessing our site ( i dont). That means we can't use seomoz then??

we'll see how this goes. I've removed the blocks for amazonaws...

Thanks .

Okay, so I couldn't find this thread and started a new one. Sorry...

... The problem persists.

RECAP

I have two blocks in my htaccess both are for amazonaws.com.

I have gone over our server block logs and see only amazon addresses and bot names.

I did a fetch as google with our WM Tools and fetch it did. Success!

Why isn't thiscrawler able to access? Many other bots are crawling right now.

Why can I use the seomoz on-page feature to crawl a single page but the automatic crawler wont access the site? Just took a break from typing this to try the on-page on our robots.txt, worked fine. Use the keyword "Disallow" and it gave me a C. =0)

... now if we could just crawl the rest of the site...

any help on this would be greatly appreciated.

I've removed all blocks but two from our htaccess. They are for amazonaws.com to block amazon from crawling us.

I did a fetch as google in our WM tools on our robots txt with success.

SEOMoz crawler here hit's our site and gets a 403. I've looks in our blocked request logs and amazon is the only one in there.

What is going on here?

Hi Keri,

Yeah, I've followed these steps. htaccess forwards to www if none and my campaign is using www.

The robots.txt is available in browser no problem.

my htaccess had massive country ip blocks as well as amazonaws..

I've removed all blocks that arent china or amazon.

Still waiting on a response from the MOZ team here on how to get the real crawl going as what I found in tools is not the same as the weekly scan.

Research Tools > Crawl Test at bottom left of page.

Not sure why it's taken me 2 days to find that but it did.

Thanks. Sorry for the dumb question...

The initial crawl only hit one page. Set up another campaign for another site and it crawled 260 pages.

How can I get the crawler started up again or do I really have to wait a week ?

I think I do. I just (a few minutes ago) went through a 403 problem being reported by another site trying access an html file for verification. Apparently they are connecting with an ip that's blocked by our htaccess. I removed the blocks told them to try again and it worked no problem. I see that SEOMoz has only crawled 1 page. Off to see if I can trigger a re-crawl now...

No problem. Looking at my Google WM Tools , crawl stats don't show any errors.

Thanks

User-Agent: *
Disallow: /*?zenid=
Disallow: /editors/
Disallow: /email/
Disallow: /googlecheckout/
Disallow: /includes/
Disallow: /js/
Disallow: /manuals/

apache

In the crawl diagnostics it says oursite.com/ has a 403. doesn't say what's causing it but mentions no robots.txt. There is a robots.txt and I see no problems. How can I find out more information about this error?

I have plugged a few KW's into the tool with a specific url. most give me a D which is kind of what I figured.

So I wanted to check something out and here are my results:

site.com/url_1 KW1 = Grade D  & Google rank is page1 pos7

site.com/url_2 KW1 = Grade A  & Google rank is nonexistant

site.com/          KW1 = Grade A  & Google rank is page2 pos6

These pages have all been alive for about 6 years and Meta stuff never changed page content has slightly over time.

Please help me understand why SEOMOZ Results != Google Results

Thank You

one keyword at a time or is there a way to use the research tools -> on page op tool for a list of keywords?

Okay. I've setup a campaign for www.site.com and given a list of keywords. So after the initial crawl we'll have some results.

What I'm looking for tho is how do individual pages on my site rank for the list of keywords given. And then be able to go to a screen in seomoz with data for that particular page with recommendations and stuff like that.

Is this what's going to happen or do I need to create a campaign for each url i want to track?

If all will work as I'd like in the example above, should I then add the second list of keywords that some other pages should rank for?

Will it get to be a big mess or can I relate the keywords to pages in some way?

It seems like what I'm looking for is what this program should be...

Thanks!