Its been over a month, rogerbot hasn't crawled the entire website yet. Any ideas?
-
Rogerbot has stopped crawling the website at 308 pages past week and has not crawled the website with over 1000+ pages. Any ideas on what I can do to get this fixed & crawling?
-
Hi TJ,
Did that answer your question? if so I am very happy. Please let me know if there's anything else I can do to assist you in fully answering this question. thank you for the kind words.
Happy holidays,
Thomas
-
@Thomas
Thanks for the headsup! Ever since asking this question Roger has crawled 2 more pages taking the total to - 310.
Yes, I've already selected Roger send details on the site on a weekly basis via the reporting section. Thanks for the quick reply and happy holidays!
Best,
TJ
-
If you have a stop at 308 that is normal Roger likes to take breaks or smaller amounts of websites in the beginning only indexes he will be back way to make sure is by following the directions below.
Have you requested that Roger check your site by going into campaigns then selecting reports then selecting weekly?
Make sure the above steps of been done. Roger is going to town on mine so I don't think you leave you out of the party.
Respectfully,
Thomas Zickell
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why don't I have permission to view the update calendar?
I am trying to figure out when the next Open Site update will be. In the past I have just gone here - https://seomoz.zendesk.com/entries/345964-linkscape-update-schedule It is saying I do not have permission even though I am logged in. Anyone else having issues? Or if anyone can see it can they see when the next update is?
Moz Pro | | EcommerceSite0 -
Still Cant Crawl My Site
I've removed all blocks but two from our htaccess. They are for amazonaws.com to block amazon from crawling us. I did a fetch as google in our WM tools on our robots txt with success. SEOMoz crawler here hit's our site and gets a 403. I've looks in our blocked request logs and amazon is the only one in there. What is going on here?
Moz Pro | | martJ0 -
Why would only 1 pg of an 18 pg site be crawled?
I signed up yesterday. Added 4 sites. The one I really need data on now has only had 1 page crawled. The other sites have had almost all pages crawled (over 40 each) in one day. What is wrong? The main domain has a 301 redirect to another domain name is that the problem? Is there something wrong at Google analytics? I dont manage this site and I'm picking up behind another firm..where should I start my discovery? Thanks so much!
Moz Pro | | moreidea0 -
Problem in doing seo for a German medical website?
Hi We have been doing seo for http://www.internetapotheke-holland.com/ since last 4 month including onpage and offpage process like link building etc. but not getting a proper output keywords are as follow: xenical holland rezeptfrei xenical Xenical Ohne Rezept Xenical Bestellen Xenical Kaufen Rezeptfrei bestellen Rezeptfrei kaufen Tramadol TropfenRezeptfrei Please could you add some more cool tips so that we can help our client. Thanks & Regards
Moz Pro | | expertvillagemedia0 -
Crawl Errors from URL Parameter
Hello, I am having this issue within SEOmoz's Crawl Diagnosis report. There are a lot of crawl errors happening with pages associated with /login. I will see site.com/login?r=http://.... and have several duplicate content issues associated with those urls. Seeing this, I checked WMT to see if the Google crawler was showing this error as well. It wasn't. So what I ended doing was going to the robots.txt and disallowing rogerbot. It looks like this: User-agent: rogerbot Disallow:/login However, SEOmoz has crawled again and it still picking up on those URLs. Any ideas on how to fix? Thanks!
Moz Pro | | WrightIMC0 -
Keyword Ranking weekly update didn't happen
A new campaign of mine is supposed to update its keyword rankings on Wednesdays but as of Thursday morning it didn't happen. I've got the original stats from the first few keywords I added when the campaign was set up, but nothing for the keywords that were added in the next day. Previous experience with this tool was pretty reliable so I'm just wondering if anyone is experiencing the same thing.
Moz Pro | | ninjaprecision0 -
Crawl Test produced only 1 page
Hi, I recently submitted a crawl for www.cirrato.com using SEOMoz Crawl Test Tool. I have a lot of pages, but the crawl result shows only 1 page, which is the front page and nothing else... Does anyone know what this could mean or what the problem is?
Moz Pro | | yusufcirrato0 -
Links listed in MozPro Crawl Diagnostics
Ok, seeing as I'm getting to the end of my first week as a Pro Member, I'm getting more and more feedback regarding the pages on my site. I'm slightly concerned though that, having logged in this morning, I'm being shown 407 warnings for pages with 'Too Many On Page Links.' According to the blurb at the top of the page, 'Too Many' is generally defined as being over 100 links on a page ... but when I look at the pages which are being thrown up in the report, none of them contain anywhere near 100 links. I seriously doubt there is a glitch with the tool which has led me to think that maybe there's an issue with the way my site is coded. Is anyone aware of a coding problem that may lead Google and SEOMoz to suspect that I have a load of links across my site? P.S. As an aside, when this tool mentions 'Too Many Links' is it referring purely to OBL or does it count links to elsewhere on my domain too? Cheers,
Moz Pro | | theshortstack0