Its been over a month, rogerbot hasn't crawled the entire website yet. Any ideas?
-
Rogerbot has stopped crawling the website at 308 pages past week and has not crawled the website with over 1000+ pages. Any ideas on what I can do to get this fixed & crawling?
-
Hi TJ,
Did that answer your question? if so I am very happy. Please let me know if there's anything else I can do to assist you in fully answering this question. thank you for the kind words.
Happy holidays,
Thomas
-
@Thomas
Thanks for the headsup! Ever since asking this question Roger has crawled 2 more pages taking the total to - 310.
Yes, I've already selected Roger send details on the site on a weekly basis via the reporting section. Thanks for the quick reply and happy holidays!
Best,
TJ
-
If you have a stop at 308 that is normal Roger likes to take breaks or smaller amounts of websites in the beginning only indexes he will be back way to make sure is by following the directions below.
Have you requested that Roger check your site by going into campaigns then selecting reports then selecting weekly?
Make sure the above steps of been done. Roger is going to town on mine so I don't think you leave you out of the party.
Respectfully,
Thomas Zickell
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SeoMoz: Competitive domain analysis report doesn't add up?
Hi, When I check OSE against the Competitive Domain Analysis report in my account the numbers are way off. If the CDA report was 100%, OSE is showing 30% of that. I'm looking at linking root domains with no filters applied. Is the data not pulled from the same db? Thanks.
Moz Pro | | Bondara0 -
Crawl started on the 18th of December and still hasn't completed!
As the title states I started a crawl over 2 weeks ago now and it still hasn't completed. Does anyone know why this could be? Thanks
Moz Pro | | cttgroup0 -
Crawled pages are missing and showing just 1 page crawled
One of my campaign has got around 8500 pages crawled(seomoz) and reports are shown, but suddenly it is showing 1 page crawled. Why it is happened like this? How can i get back the previous reports?
Moz Pro | | Sulekha0 -
Can you set-up a manual SEOmoz crawl?
I received a crawl report yesterday, made some site changes, and would like to see if those changes were done correctly. Rather than wait a week for my automatic crawl to be generated, is there anyway to initiate a manual crawl on a single subdomain as a PRO member? As a PRO member, you can schedule crawls for 2 subdomains every 24 hours, and you'll get up to 3,000 pages crawled per subdomain. When we've finished crawling, your reports will be sent to your PRO email address, which is currently From here... http://pro.seomoz.org/tools/crawl-test
Moz Pro | | ICM0 -
Canonical tags and SEOmoz crawls
Hi there. Recently, we've made some changes to http://www.gear-zone.co.uk/ to implement canonical tags to some dynamically generated pages to stop duplicate content issues. Previously, these were blocked with robots.txt. In Webmaster Tools, everything looks great - pages crawled has shot up, and overall traffic and sales has seen a positive increase. However the SEOmoz crawl report is now showing a huge increase in duplicate content issues. What I'd like to know is whether SEOmoz registers a canonical tag as preventing a piece of duplicate content, or just adds to it the notices report. That is, if I have 10 pages of duplicate content all with correct canonical tags, will I still see 10 errors in the crawl, but also 10 notices showing a canonical has been found? Or, should it be 0 duplicate content errors, but 10 notices of canonicals? I know it's a small point, but it could potentially have a big difference. Thanks!
Moz Pro | | neooptic0 -
Is there a whitelist of the RogerBot IP Addresses?
I'm all for letting Roger crawl my site, but it's not uncommon for malicious spiders to spoof the User-Agent string. Having a whitelist of Roger's IP addresses would be immensely useful!
Moz Pro | | EricCholis1