Rogerbot's crawl behaviour vs google spiders and other crawlers - disparate results have me confused.
-
I'm curious as to how accurately rogerbot replicates google's searchbot
I've currently got a site which is reporting over 200 pages of duplicate/titles content in moz tools. The pages in question are all session IDs and have been blocked in the robot.txt (about 3 weeks ago), however the errors are still appearing.
I've also crawled the page using screaming frog SEO spider. According to Screaming Frog, the offending pages have been blocked and are not being crawled. Webmaster tools is also reporting no crawl errors.
Is there something I'm missing here? Why would I receive such different results. Which one's should I trust? Does rogerbot ignore robot.txt? Any suggestions would be appreciated.
-
Thanks for your response. I was beginning to think this question had been left to rot.
I'm not getting any errors in WMT. What is concerning is that Roger is returning almost 300 errors of dupe content, which is obviously a problem. Screaming frog is no longer finding the pages (they've been blocked in the robot.txt) I guess what I'm trying to ask here is how can I be sure that my dupe content has been effectively blocked from google's spider.
Is there anyway to check?
Thanks for your help.
-
I've see similar concerns from others, it seems "rogerbot" does ignore certain things that other bots consider.
Don't worry about it, if it's not being flagged in WMT it shouldn't be an issue.
Take Roger as a guide rather than an iron fist bot like googlebot.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can't disconnect or change Google Analytics profile
I would like to change the Google Analytics profiles for my SEOmoz campaigns. But every time I try to disconnect or change profile, the request timeout. What to do?
Moz Pro | | Bownty0 -
Confused on setting up my domain in a campaign
I have a website that is www.example.com. When I first setup my campaign in SEOMoz, I missed the instructions that stated I need to set up my domain as it appeared in the browser. Thus, I set it up as example.com. When I realized my mistake, I setup a second campaign as www.example.com and am running them side by side to see the differences. My question has to do with the best setup. I have a blog that is a subdomain as blog.example.com. It appears that the www.example.com campaign is not crawling the blog pages but example.com is. Are there any downsides in setting up this campaign as example.com? I have a number of other sites with the same issue so would like to get this resolved before I setup the additional sites. Thanks for any assistance or insights into ways I may be "shooting myself in the foot" with the wrong domain structure in my campaign settings.
Moz Pro | | rfwood0 -
Crawl Diagnostics Update
I have corrected some errors in my SEOMoz Crawl Diagnostics, however the errors are still showing. It says a crawl has happen since. Any idea's why?
Moz Pro | | petewinter0 -
How to tell where a competitor's Facebook Shares are coming from?
Is there a way to tell where a competitor's facebook shares and likes are coming from? How to also tell what ads they have running in Facebook and their spend? I am looking at OSE's data and its telling me a competitor has 8.5K Facebook shares and 1K facebook likes. I go to their facebook page and it has about 25 likes. This site should not have anywhere close to the facebook shares/likes its receiving so wondering where they are getting their boosted traffic from.
Moz Pro | | rjb6270 -
Crawl Diagnostics Summary
Is there a way to view the charts in the crawl diagnostics summary on a monthly view (or export the monthly figures)?
Moz Pro | | RikkiD220 -
What's the best & quick tool to spy or analyze backlinks of SINGLE Page?
hey, someone I want to know why this page or blog post of my competition website ranking better.. I've many tools to spy & analyze domain name, but personalty I want to spy just one page of this or that website or blog.. I hope you could help me and share some tools or the best once for this job. thnx again
Moz Pro | | akitmane0 -
Crawl complete, but nothing changed?
Hi everyone, According to my account, the crawl diagnostics were completed yesterday. However, the duplicate page titles that it mentions aren't correct. The changes that I implemented several days ago are not being shown in the report. When I click the duplicate page title links, the latest date in the graph is 3/26. However, it says the crawl was completed on 3/30. Does it take a few days for the reports to match what the crawl actually discovered?
Moz Pro | | beeneeb0 -
What's Your SEOmoz Favorite?
Obviously, SEOmoz is full of awesomeness. But I'm curious as to what sections, tools, etc. of SEOmoz are the users' favorites? For me, it's a close race between the blog, Linkscape/Open Site Explorer, and the search overlay tool on the Mozbar. What are your favorites?
Moz Pro | | Gyi0