Welcome to the Q&A Forum

Hi Peter

thanks for your reply. The crawl has now completed and given me some more areas to work on, it's a great tool.

I was so preoccupied with 'hiding' the site over the last couple of months with the easy code:

User-agent: *
Disallow: /

I hadn't thought beyond this.

I've noticed Google has now recognised the new robots.txt which has allowed the sitemap to be accepted..

I'll look at your notes, thank you, and work out my next move. I'll let you know how I get on too.

I know (well think) I have to get noindex, follow for 'sorted' category pages...

all the best, Mark.

Hi Mike

The crawl has now completed, thank you. I think the results will keep me occupied

all the best, Mark.

Hi all, I've noticed Roger bot is taking a long time to crawl my new site. It started on the 28th Feb 2013 and is still going. There aren't many pages at the moment. Any ideas please?

thanks a lot, Mark.

Hi all, I've noticed Roger bot is taking a long time to crawl my new site. It started on the 28th Feb 2013 and is still going. There aren't many pages at the moment. Any ideas please?

thanks a lot, Mark.