...actually I take that back. Still reporting as blocked by robots.txt.
Going to email the team.
Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
Job Title: Director
Company: Ignician
...actually I take that back. Still reporting as blocked by robots.txt.
Going to email the team.
Thanks, it appears to be crawling without issue now
I'm in the process of working with a site under development and wish to run the SEOmoz crawl test before we launch it publicly. Unfortunately rogerbot is reluctant to crawl the site. I've set my robots.txt to disallow all bots besides rogerbot.
Currently looks like this:
User-agent: *
Disallow: /
User-agent: rogerbot
Disallow:
All pages within the site are meta tagged index,follow.
Crawl report says:
Search Engine blocked by robots.txt Yes
Am I missing something here?
10 years experience in web design, development, ecommerce and SEO.
Looks like your connection to Moz was lost, please wait while we try to reconnect.