Rogerbot's crawl behaviour vs google spiders and other crawlers - disparate results have me confused.
-
I'm curious as to how accurately rogerbot replicates google's searchbot
I've currently got a site which is reporting over 200 pages of duplicate/titles content in moz tools. The pages in question are all session IDs and have been blocked in the robot.txt (about 3 weeks ago), however the errors are still appearing.
I've also crawled the page using screaming frog SEO spider. According to Screaming Frog, the offending pages have been blocked and are not being crawled. Webmaster tools is also reporting no crawl errors.
Is there something I'm missing here? Why would I receive such different results. Which one's should I trust? Does rogerbot ignore robot.txt? Any suggestions would be appreciated.
-
Thanks for your response. I was beginning to think this question had been left to rot.
I'm not getting any errors in WMT. What is concerning is that Roger is returning almost 300 errors of dupe content, which is obviously a problem. Screaming frog is no longer finding the pages (they've been blocked in the robot.txt) I guess what I'm trying to ask here is how can I be sure that my dupe content has been effectively blocked from google's spider.
Is there anyway to check?
Thanks for your help.
-
I've see similar concerns from others, it seems "rogerbot" does ignore certain things that other bots consider.
Don't worry about it, if it's not being flagged in WMT it shouldn't be an issue.
Take Roger as a guide rather than an iron fist bot like googlebot.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawlers reporting upper case letter url versions although these have been 301'd to lower case !?
Hi I have a client e-com site who's dev platform is on a windows server Their product pages have been auto-named after the product title, with the first letter in each word being upper case, which has hence translated to the URL having upper cases instances too. I asked them to set up 301 redirects for all url's that had upper case instances to lower case versions, which they say they have done. However I'm still seeing url's with upper case instances showing up in webmaster tools and moz crawl reports but when I copy & paste them into a browser they do redirect to, & resolve in, the lower case version. Its also upper case versions reported in the Google cache! So how come webmaster tools & Moz etc are reporting the upper case versions, surely if redirected it should be the lower case versions All Best Dan
Moz Pro | | Dan-Lawrence0 -
Difference between Open site Explorer's Root Domain and Basic SERP Report's Linking Root Domain?
Why show different Linking Root Domain open site explorer and SERP of any websites? Open Site explorer show different linking root domain and Basic SERP Report show different linking root domain of any website url, who is the correct and why it is show different linking root domain?
Moz Pro | | surabhi60 -
Campaign Crawl
I have a site with 8036 pages in my sitemap index. But the MozBot only Crawled 2169 pages. It's been several months and each week it crawls roughly the same number of pages. Any idea why I'm not getting fully crawled?
Moz Pro | | JMFieldMarketing0 -
My Campaign has been crawling for about a week now
Can anyone tell me why one of my campaigns has been stuck in crawl mode for about a full week and it is still not done?!?!
Moz Pro | | nazmiyal0 -
Does Open Site Explorer violate Google's Terms of service?
According to Google's Webmaster Guidelines: "Don't use unauthorized computer programs to submit pages, check rankings, etc. Such programs consume computing resources and violate our Terms of Service." Does that mean Open Site Explorer is a violation of those Terms of Service, or is it authorized?
Moz Pro | | ericwagner0 -
RogerBot does not respect some rules??
Hello; Every week when I see my stats I notice that RogerBot has crawled 10000 form my website, even pages with a no index or not allowed in the robots.txt. Is it possible to avoid him from crawling the these pages? They are form pages in my site, with are not indexed by google, they have a noindex and they are not allowed for crawling in the robots.txt. Thanks everyone for your help!!!
Moz Pro | | jgomes0 -
Urgent: Campaign set up 'Select Competitors' errors
Hi. Im setting up my first campaign and Im having issues with step 3: 'Select your competitors to track'. I only want to track 1 competitor: http://en.wikipedia.org/wiki/Ryan_Murphy_(writer) When I enter this and the competitor name into the form provided and click 'continue to next step' it throws an error at me: Darn, there are errors in your form! Don’t worry, Roger can’t feel pain. Competitors domain http://en.wikipedia.org/wiki/ryan_murphy_(writer) may not have a /path after the host Domain http://en.wikipedia.org/wiki/ryan_murphy_(writer) may not have a /path after the host Can anyone help me as this is urgent.
Moz Pro | | RyanSMurphy1 -
SEOmoz Toolbar vs. Opensiteexplorer
Dumb question, why is the SEOmoz Toolbar reporting vastly different data than opensitexplorer? I had assumed they pulled from the same data set. False assumption? Am I misinterpreting the metrics? The discrepancies with which I am most confused are differences in number of root linking domains between OSE and Toolbar. Please enlighten me.
Moz Pro | | Gyi0