Why doesn't Moz crawler follow robots.txt?
-
It is crawling the entire site, and there is stuff we do not want it to. Please advise.
-
Which I am ok with, but why am I getting duplicate content?
-
Yes, it doesn't tell them which pages not to crawl - just not to index them
-
It has been used correctly. The site is a Magento site and they have it built in. There are a lot of filters for products so it uses rel=canonical to tell Google which to index.
-
rel=canonical is not really an robots instruction file - rel=canonical is to help with duplicate copy where you have the same or similar pages and your telling search engines which pages is the preferred page.
If you don't want pages crawling you have to tell Search engines in the robots file
-
Hi There,
Rel=canonical tags tell robots, which page is actually to index out of many.
For SEOs, canonicalization refers to individual web pages that can be loaded from multiple URLs. This is a problem because when multiple pages have the same content but different URLs, links that are intended to go to the same page get split up among multiple URLs. This means that the popularity of the pages gets split up. Unfortunately for web developers, this happens far too often because the default settings for web servers create this problem.
https://moz.com/learn/seo/canonicalization
I feel you have not used it correctly, check the above article and see if it helps.
Thanks,
Vijay
-
So I made a mistake it isn't the robots.txt that is the issue. I am getting hit with a ton of duplicate content penalties so I figured that was it. The problem is that I have pages with rel=canonical tags that it is ignoring. Does Roger not read those?
-
Hi
Have to agree with the above, Rogerbot does listen to robot.txt file, unlike Bing - while they are getting better Bing ignores the robots.txt file frequently.
Ive analysed quite a few server logs over the years and Roger has always listened to the file - its usually a mistake the in the robots file.
There is an option to test your robots.txt file in GCS - while this is testing to see if Google will crawl the page - usually Roger has the same instructions as Google.
However if you are still pretty certain that Roger is ignoring robots.txt please DM your Server Logs and your website and I will take a look and analyse it for you (free of course).
Thanks
Andy
-
All major search engines, including Moz's crawler Rogerbot and Internet Archives, respect Robots.txt as a standard “robots exclusion protocol” to communicate with web crawlers and web robots.
In case you wish to exclude some specific information from all Search Engines, you can use the following sample code as reference to block specific directories.
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/However, if you want to specifically block Mz's Rogerbot from crawling specific sections of your website. You may take the following reference code to block specific areas / directories in your website from rogerbot:
User-agent: Rogerbot
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/I hope this helps, If you have specific questions, please feel free to respond, I will be happy to answer them.
Regards,
Vijay
-
Hi there! Moz's crawler, rogerbot, does follow robots.txt. When he's not following robots.txt, it's usually because the robots.txt protocol is formatted improperly. Learn more about formatting your page here: https://moz.com/learn/seo/robotstxt
For more information on Roger, including how to block him, head here: https://moz.com/help/guides/moz-procedures/what-is-rogerbot
And if you want to test your formatting, try the Robots Checker here: https://support.google.com/webmasters/answer/6062598
If you're still unable to determine why rogerbot is crawling your site, feel free to write in to help@moz.com!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I couldn't check Page authority or Domain Authority for any page, what can be reason?
I am trying to check DA and PA for page but not able to do this. Tried it before two days and it was working fine, problem occurring today. I want help regarding this issue, is anyone other facing this problem. Kindly help me out (provide some other way to check DA/PA).
Link Explorer | | JessicaThompson1 -
Does Moz register backlinks from Twitter/Facebook/LinkedIn?
I've noticed that on sites with decent social presence and regular content posts that Moz isn't showing/recording any inbound links from Twitter, Facebook or LinkedIn. Posts are regularly shared via these three channels but no inbound links ever seem to show up. Does Moz not record such inbound links, or are they all disguised using JavaScript or some other trick?
Link Explorer | | JCN-SBWD0 -
Why Moz Link Explorer is showing wrong data?
Hi everyone. I am using Moz Link Explorer to find relevant links from competitors and when I narrow it down to Link Type: Follow Status: Active I get the list. All good here... BUT ... Inspecting the actual link on the page there is a no=follow attached to it. Is this feature still working properly on the new Link Explorer or is a bug? Thanks!
Link Explorer | | Sorin_T0 -
In "Link Opportunities" is there a way to filter by external follow links that pass equity?
In Open Site Explorer, the "Link Opportunities" feature does not seem to have a way to filter for external follow links that pass equity. This would be very useful. Is this a feature in the pipeline or is there already a way to do this?
Link Explorer | | GlennFerrell2 -
Does Moz allow you to upload your disavow to take into account when analysing links?
I was wondering if moz allows you to upload your disavow file and if it has a consideration in the data supplied? Otherwise how can we accurately use moz to understand the true analytics of our site?
Link Explorer | | MoneySite0 -
Open Site Explorer Doesn't Discover Link for my website
Hi, I've been trying to use Open Site Explorer to track link to my own website but it always comes up with no data and shows me this message: It looks like we haven't discovered link data for this site or URL. However, this site has been up and running for more than 9 months and google webmaster can track all the links to my website. I've also followed MOZ's support instruction to tweet a bit.ly links with my website but still no luck at all. My website is www.carpetcleanerperthwa.com.au Has anyone seen the same problem before? Any help would be much appreaciated. Thanks Will
Link Explorer | | willwai0 -
Moz crawling bot
Hi guys, in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt". I tried to find out what the ID of Moz crawling bot is, and on this page: http://moz.com/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot). However, my robots.txt doesn't have such entry. Take a look: Automatically banned scanners and crawlers section User-agent: 008 Disallow: / user-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: metajobbot Disallow: / User-agent: Exabot Disallow: / User-agent: Ezooms Disallow: / User-agent: fyberspider Disallow: / User-agent: dotbot Disallow: / User-agent: MojeekBot Disallow: / Section end What could be the problem here, then? Why does the Moz bot think I'm blocking it?
Link Explorer | | superseopl0 -
Why does OSE & Moz Q&A sections not work for me in Chrome any more?
A few weeks ago I started seeing instances of OSE not showing all the information when it was showing I was logged in while using Google Chrome. It would work, then not work and kept showing me the button to sign up for Moz Analytics even though I was logged in. I'm a Pro Member, which is confusing. Then I noticed that while logged in, I would be logged out somehow when trying to contribute to the Q&A's and it would bounce me back to the Moz Home page or show me the white Login button in the upper right. Again, in Chrome only. I was told by support to clean my cookies/cache and that works for maybe a few hours, but then would resort back to logging me out. It has become very annoying, and now can only work logged in from FireFox. Not a huge issue, but everything else I do all day is in Chrome. Is anyone else having this same issues? Any recommendations? Thanks in advance. - Patrick
Link Explorer | | WhiteboardCreations0