...actually I take that back. Still reporting as blocked by robots.txt.
Going to email the team.
Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
...actually I take that back. Still reporting as blocked by robots.txt.
Going to email the team.
Thanks, it appears to be crawling without issue now
I'm in the process of working with a site under development and wish to run the SEOmoz crawl test before we launch it publicly. Unfortunately rogerbot is reluctant to crawl the site. I've set my robots.txt to disallow all bots besides rogerbot.
Currently looks like this:
User-agent: *
Disallow: /
User-agent: rogerbot
Disallow:
All pages within the site are meta tagged index,follow.
Crawl report says:
Search Engine blocked by robots.txt Yes
Am I missing something here?