Rogerbot did not crawl my site ! What might be the problem?
-
When I saw the new crawl for my site I wondered why there are no errors, no warning and 0 notices anymore.
Then I saw that only 1 page was crawled. There are no Error Messages or webmasters Tools also did not report anything about crawling problems.
What might be the problem?
thanks for any tips!
Holger -
I am running into this problem as well. Does anyone have an answer. I was a customer a year ago and had no problems. Stopped for a while and then came back, my site was never able to be crawled again. A few months later I did make a few changes on my site, and just tried again today, but it only gets 1 page. Once before they told it had to do with something like http 1.0/1.1 which makes no sense. Not sure why google, bing and the rest can crawl the site, but this service can't. Hopefully someone has an answer. I looked at that link that is mentioned and I do not feel that any of those are causing the issues.
-
Hi there,
Thanks for reaching out! I went ahead and started a ticket for you to do some digging. I will run a few tests on my end to see where it is causing the issue. I will get in touch you ASAP
Thanks again for reaching, hope I could talk to you soon!
Best,
-
Thanks Keri,
But its not about these common problems, because Rogerbot crawled the properly before. May be a ticked will help to find the cause.
Holger
-
Hi! Here are some common reasons why only one page is crawled. If this doesn't solve it, opening a ticket with the help desk would be the next step.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz crawl duplicate pages issues
Hi According to the moz crawl on my website I have in the region of 800 pages which are considered internal duplicates. I'm a little puzzled by this, even more so as some of the pages it lists as being duplicate of another are not. For example, the moz crawler considers page B to be a duplicate of page A in the urls below: Not sure on the live link policy so ive put a space in the urls to 'unlive' them. Page A http:// nuchic.co.uk/index.php/jeans/straight-jeans.html?manufacturer=3751 Page B http:// nuchic.co.uk/index.php/catalog/category/view/s/accessories/id/92/?cat=97&manufacturer=3603 One is a filter page for Curvety Jeans and the other a filter page for Charles Clinkard Accessories. The page titles are different, the page content is different so Ive no idea why these would be considered duplicate. Thin maybe, but not duplicate. Like wise, pages B and C are considered a duplicate of page A in the following Page A http:// nuchic.co.uk/index.php/bags.html?dir=desc&manufacturer=4050&order=price Page B http:// nuchic.co.uk/index.php/catalog/category/view/s/purses/id/98/?manufacturer=4001 Page C http:// nuchic.co.uk/index.php/coats/waistcoats.html?manufacturer=4053 Again, these are product filter pages which the crawler would have found using the site filtering system, but, again, I cannot find what makes pages B and C a duplicate of A. Page A is a filtered result for Great Plains Bags (filtered from the general bags collection). Page B is the filtered results for Chic Look Purses from the Purses section and Page C is the filtered results for Apricot Waistcoats from the Waistcoat section. I'm keen to fix the duplicate content errors on the site before it goes properly live at the end of this month - that's why anyone kind enough to check the links will see a few design issues with the site - however in order to fix the problem I first need to work out what it is and I can't in this case. Can anyone else see how these pages could be considered a duplicate of each other please? Checking ive not gone mad!! Thanks, Carl
Moz Pro | | daedriccarl0 -
Does SEOmoz have a tool to find mirror sites?
I heard from a company that is trying to get my clients SEO business that they discovered multiple sites mirroring our site's content. Does SEOmoz have a tool to find these websites? Or does Google?
Moz Pro | | thomas.wittine0 -
Does crawling help in optimisation.?
the website is as it was last week. no optimisation from my side for 10 days now. i was ranked 5 with my keyword not much competition there. however 2 days ago i registrred at seomoz and created a campaign for my website with my keywords that were ranked 5 in search. today i see that my rank has gone up to 2. i have nt done any optimisation neither have ii created any backlinks. so how and why did i climb up? i just created a campaign and let seomoz crawl my website for 2days. am i to assume seomoz crawl optimises website? if that is the case then can i create a campaign crawl pages, climb up in searches, delete the campaign after a week, create it again crawl pages and climb up and so on ? please advise?
Moz Pro | | wahin10 -
A question about Mozbot and a recent crawl on our website.
Hi All, Rogerbot has been reporting errors on our website's for over a year now, and we correct the issues as soon as they are reported. However I have 2 questions regarding the recent crawl report we got on the 8th. 1.) Pages with a "no-index" tag are being crawled by roger and are being reported as duplicate page content errors. I can ignore these as google doesnt see these pages, but surely roger should ignore pages with "no-index" instructions as well? Also, these errors wont go away in our campaign until Roger ignores the URL's. 2.) What bugs me most is that resource pages that have been around for about 6 months have only just been reported as being duplicate content. Our weekly crawls have never picked up these resources pages as being a problem, why now all of a sudden? (Makes me wonder how extensive each crawl is?) Anyone else had a similar problem? Regards GREG
Moz Pro | | AndreVanKets0 -
Problem with Rankings and On-page Optimization
Hi SEOZ 🙂 I have a question regarding the Rankings and On-page Optimization in the seomoz Campaign Manager. I have setup a website url for examle: www.keyword.com After that created a list of all the target Keywords, that I want to reach within my Website: keyword-a, keyword-b, keyword-c and so on... Then I did a On-Page Analysis for all the urls with the specific keywords. keyword-a: www.keyword.com/keyword-a/ keyword-b: www.keyword.com/keyword-b/ keyword-c: www.keyword.com/keyword-c/ and so on... Most of the urls got the A grade. Now after the Website hast launched and got crawled, I have a problem with the Rankings and the On-page Optimization. The Rankings for my Keywords and also the Grades at the On-page Optimization are only shown for my Start/Homepage: www.keyword.com NOT for the urls that are specific for a Keyword for example: www.keyword.com/keyword-a/ Also the Grades are shown for the Keywords but again only in combination with my Start/Homepage www.keyword.com NOT for www.keyword.com/keyword-a/ What is the problem? Bye, Alex
Moz Pro | | krseo0 -
Crawl Diagnostics Report
I'm a bit concerned about the results I'm getting from the Crawl Diagnostics Report. I've updated the site with canonical urls to remove duplicate content and when I check the site - it all displays the right values, but the report, which has just finished crawling is still showing a lot of pages as duplicate content. Simple example: http://www.domain.com http://www.domain.com/ Both of them are in the duplicate content section although both have canonical url set as: Does each crawl check the entire site from the beginning or just the pages it didn't have a chance to crawl the last time? This is just one of 333 duplicate content pages, which have canonical url pointing to the right page. Can someone please explain?
Moz Pro | | coremediadesign0 -
SEOMoz Pro and Multiple Niche Sites
Currently I have 15 affiliate marketing sites and three blogs which are monetized. My budget dictates the base pro package so may I get some recommendations on how or if I can do analysis over time for all the sites. I ask this because it seems you would set your campaigns for long-term but I imagine you delete/modify/add campaigns to analyze different key words. Thanks for your help.
Moz Pro | | JavaManOne0
