Why does SEOMoz crawler ignore robots.txt?
-
The SEOMoz crawler ignores robots.txt
It also "indexes" pages marked as noindex.
That means it is filling up the reports with things that don't matter.
Is there any way to stop it doing that?
-
Hi Alan,
The code should be ok
Try to "drive-test" it with a custom crawl from http://pro.seomoz.org/tools/crawl-test then you will see if it works well.
I am glad the link was useful.
Gr.,
Istvan
-
Thank you István
I added this:
User-agent: rogerbot
Disallow: /sendtoafriend/
Disallow: /photo/
Disallow: /pix/- because crawlers shouldn't go down those paths and roger is detecting pages without descriptions.
Is what I added OK?
-
Hi,
You can block RogerBot from Robots.txt
Check for further instructions on: http://www.seomoz.org/dp/rogerbot
"Please note: Adding this code will prevent our crawl test tool from being able to crawl your website."
Gr.,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Why does Moz Crawler start with HTTP//1.1 version??
We have run Moz Pro's Site Crawl for example-domain.com Why has Moz's crawler decided to site http://www.example-domain.com (ie the non-secure version) as zero crawl depth page and report the actually live https://www.example-domain.com (ie secure HTTP2 version) as a crawl depth of 2? Surely the main live page should be the first crawled and reported as crawl depth = 0?
Moz Pro | | AKCAC0 -
No follow links also been reported in SEOmoz crawl diagnostics
Hi, Why does SEOmoz reports links which has been marked as 'nofollow'. I am getting 'Overly-Dynamic URL' reports on links which I have designated as nofollow which means Google will discount them. So why does SEOmoz still report them. Thanks.
Moz Pro | | malpani0 -
How does SEOmoz pull its duplicate page title and content information?
I ask because I am getting errors based on URLs that do not even exist on our site. For example: http://www.robots.com/applications/abb/panasonic/robots this URL does not even exist for our site, but somehow it is listed in the error section of page title duplication tool. http://www.robots.com/applications/ exists, but there is no place to get to an ABB or a Panasonic robot from this page, not to mention an ABB/Panasonic (which for sure does not exist). ?? We have quite a few of these out there and just wondering how to find out where the link is coming from. When we checked our URLs through Integrity, links like the one listed above (which we had 29 of them listed) that do not show up. Thoughts? Thanks! Janelle
Moz Pro | | jwanner0 -
SEOmoz ranking report SERP
I was wondering how SEOmoz the SERPs tracks? e.g. the SERP of keyword in the Google US report, doesn't give me the same result as https://www.google.com/search?pws=0&gl=us&q=keyword Does SEOmoz check the google.com SERPs from several locations and calculate an average?
Moz Pro | | Teklan0 -
SEOMOZ Crawler unicode bug
for the last couple of weeks the SEOMOZ crawls my homepage only and gets 4xx error for most of the URL's. the crawler have no issues with English url's only with the unicode(Hebrew) ones. this is what is see in the csv export for the crawl (one sample) : http://www.funstuff.co.il/׳ž׳¡׳™׳‘׳×-׳¨׳•׳•׳§׳•׳× 404 text/html; charset=utf-8 you can see that the URL is Gibberish please help.
Moz Pro | | AsafY0 -
SEOMOZ tools are not helping!
So, I have a website www.mobikwik.com which provides mobile phone topup in India. I signed up for SEOMOZ Pro membership a little less than a month ago. For all our competitors , which rank above us in google as of today , I checked out two things a) competitive link analysis and b) on-page optimization. On b) , our home page had the best optimization for targeted keywords and the best score amongst all competitors who ranked above us. On a) , linkscape tools showed our moztrust and mozrank to be the best. Except for 2 competitors, our domain authority was also better than the rest. Inspite of all this, our website is ranked no more than 6 on google for all targeted keywords. We have the best quality links amongst our competitors. I am at a loss to understand this and how to improve our ranking. I have to make a decision whether to renew my SEOMOZ membership in 2 days time. Please help me decide!
Moz Pro | | mobikwik0 -
How do i get to know th pages crawled by SEOMOZ?
My SEOMOZ campaign says that "n" number of pages were crawled. How do i get access to the list of the pages crawled by SEOMOZ?
Moz Pro | | IM_Learner0 -
Seomoz staff help needed
I have this new client new domain new website, www.ard.uk.com when seo moz crawls the site it says it has 24 million links mozrank of over 7, looks amazing but impossible can someone as seomoz have a look please
Moz Pro | | francesco-2850160