Why does SEOMoz crawler ignore robots.txt?
-
The SEOMoz crawler ignores robots.txt
It also "indexes" pages marked as noindex.
That means it is filling up the reports with things that don't matter.
Is there any way to stop it doing that?
-
Hi Alan,
The code should be ok
Try to "drive-test" it with a custom crawl from http://pro.seomoz.org/tools/crawl-test then you will see if it works well.
I am glad the link was useful.
Gr.,
Istvan
-
Thank you István
I added this:
User-agent: rogerbot
Disallow: /sendtoafriend/
Disallow: /photo/
Disallow: /pix/- because crawlers shouldn't go down those paths and roger is detecting pages without descriptions.
Is what I added OK?
-
Hi,
You can block RogerBot from Robots.txt
Check for further instructions on: http://www.seomoz.org/dp/rogerbot
"Please note: Adding this code will prevent our crawl test tool from being able to crawl your website."
Gr.,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt
I have a page used for a reference that lists 150 links to blog articles. I use in in a training area of my website. I now get warnings from moz that it has too many links. I decided to disallow this page in robots.text. Below is the what appears in the file. Robots.txt file for http://www.boxtheorygold.com User-agent: * Disallow: /blog-links/ My understanding is that this simply has google bypass the page and not crawl it. However, in Webmaster Tools, I used the Fetch tool to check out a couple of my blog articles. One returned an expected result. The other returned a result of "access denied" due to robots.text. Both blog article links are listed on the /blog/links/ reference page. Question: Why does google refuse to crawl the one article (using the Fetch tool) when it is not referenced at all in the robots.text file. Why is access denied? Should I have used a noindex on this page instead of robots.txt? I am fearful that robots.text may be blocking many of my blog articles. Please advise. Thanks,
Moz Pro | | Rong
Ron0 -
Does SEOmoz have a Keyword Research tool similar to, say, the Google AdWords tool or the WebCEO Keyword Research Tool? And where might that be? (Sorry, I'm very new to SEOmoz Pro.)
I'm looking for an SEOmoz version of the classic WebCEO Keyword Research that would give you effective suggestions based on a keyword inquiry. I've made the switch from WebCEO, but I'm trying to find something similar to that Keyword Research tool. Am I going to just need to use the Google AdWords tool for this function or does SEOmoz have it's own version?
Moz Pro | | SmokewagonKen0 -
Why my seomoz keyword ranking are different to google SERPs
I use Cute Rank to check my keywords ranking in Google, but after last update seomoz is showing some of keyword ranked much higher than Cute Rank. I checked them manually and they are on positions showed by Cute Rank and not by Seomoz. Can someone explain me why is that?
Moz Pro | | misterdam0 -
How to increase ranking on google by using pro seomoz
Hi Team, I am trying to get good rank on this 2 keywords, gift card http://www.giftbig.com/gift-cards.html and gift vouchers http://www.giftbig.com/gift-vouchers.html, can anyonw suggest me how to rank on these 2 keywords. Now I m using SEO moz pro to get the solution.
Moz Pro | | Joydeep_das0 -
Does SEOMoz ever work?
Hi, I've signed up for the free 30 day trial and I'm on the edge of not actually subscribing to the service. I go through the Q & A boards which I find really interesting and hope I can add value there in the future, but the tools interest me more (and this is where the issue lies. Do they ever work? The Keyword difficulty tool just constantly says to come back in 20 minutes and I don't think the Rank Tracker has worked for at least half my freebie 30 days. Have the tools always been this flaky or is it a blip?
Moz Pro | | orlandovisiting1 -
SEOMoz Q&A having some issues?
Apologies if this has been asked. I was browsing Q&A and got a Ruby error page, reloaded and everything was fine. However a bunch of the topics are kind of messed up, and it looks like a chunk of them are missing now. For example my 'questions I've answered' section of My Q&A only has the newest question and the oldest question, and when I sort the SEOMoz Tools category by newest I get one recent post and then the next one is from November of last year. You guys having database problems?
Moz Pro | | icecarats0 -
Why is the SEOmoz customer service on this site so awful?
Half the time the tools don't work and there isn't any accountability. Has anyone else had the same experience?
Moz Pro | | wpsoule1