Why does SEOMoz crawler ignore robots.txt?
-
The SEOMoz crawler ignores robots.txt
It also "indexes" pages marked as noindex.
That means it is filling up the reports with things that don't matter.
Is there any way to stop it doing that?
-
Hi Alan,
The code should be ok
Try to "drive-test" it with a custom crawl from http://pro.seomoz.org/tools/crawl-test then you will see if it works well.
I am glad the link was useful.
Gr.,
Istvan
-
Thank you István
I added this:
User-agent: rogerbot
Disallow: /sendtoafriend/
Disallow: /photo/
Disallow: /pix/- because crawlers shouldn't go down those paths and roger is detecting pages without descriptions.
Is what I added OK?
-
Hi,
You can block RogerBot from Robots.txt
Check for further instructions on: http://www.seomoz.org/dp/rogerbot
"Please note: Adding this code will prevent our crawl test tool from being able to crawl your website."
Gr.,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No follow links also been reported in SEOmoz crawl diagnostics
Hi, Why does SEOmoz reports links which has been marked as 'nofollow'. I am getting 'Overly-Dynamic URL' reports on links which I have designated as nofollow which means Google will discount them. So why does SEOmoz still report them. Thanks.
Moz Pro | | malpani0 -
Seomoz crawler problems
I have had Seomoz for about a month. It has crawled about 1000 pages. I have about 10,000 pages total for the site. Why are these others being a problem? I have contacted help but the guy isn't any help, we have just been going back and forth for the last two weeks. Any suggestions?
Moz Pro | | EcommerceSite0 -
Magento, SEOMOZ Toolbar not showing my HTML text, just shows CData code...
When I use the Mozbar on my homepage http://bootsjeansandleathers.com it just shows: // where it should show the HTML text of the page....anyone any ideas? Does this mean Google doesn't read my HTML text on the page!? Many thanks! mozbar.png
Moz Pro | | ebowdublin0 -
SEOmoz campaign - keyword rankings...
Hi there, I have a quick question: Does it make a difference if I add keywords to a campaign with the '[' and ']' brackets included? I just realized that I had copy/pasted many of my keywords from the exported CSV from the Google AdWords Keyword Tool and I fear that the keyword rankings on SEOmoz might factor the brackets as part of the keywords themselves. Thanks in advance! Ben
Moz Pro | | BenHammer0 -
Edit SEOmoz Profile Bug
Hi,
Moz Pro | | rayvensoft
I am trying to complete my SEOmoz profile, but there seems to be a bug. I am able to fill out everything except the 'favorite topics'. It lets me go in and fill it out, but when I click the update button to save it it does not save, and still shows up as missing in bar at the top. I have tried it in Mozilla, IE and Chrome, and it still does not work. Am I missing something? Thanks.0 -
Do you think Seomoz is worth the monthly fee if you're not a professional SEO ?
I just want to ask the people who subscribe to Seomoz on a regular basis, I just paid for my first month subscription but to be perfectly honest I'm trying to work out whether
Moz Pro | | whitbycottages
somebody who is not rolling in cash and trying to make a living can afford to
pay the fee each month. I'm not a professional I just have two business websites and I'm learning the subject and finding it interesting. The tools do seem very good but I just wondered how people see this service on which aspect is the most important of them. I like to continue, I have been impressed the quality of the forum topics and discussions I just wonder whether I can afford to justify the fee.1 -
Is the SEOMoz Rank Tracker still accurate?
I monitor several keywords on several domains and i have found that rank tracker gives me one result yet when searching via Google myself (on different machines) i get a different result. I've tried several devices on different networks so there is no chance that Google is remembering my searches as such and i ensure that i am always signed out of my account. All devices provide the same results yet Rank Tracker provides a completely different result. It's as though Rank Tracker is a day ahead or something. Can anyone shed any light for me please?
Moz Pro | | MarkHincks0 -
Why am I still Unranked in SEOmoz?
I was very active in SEOmoz's Q&A forums today and completed my profile - raising my Moz points from 0 to 50. On my profile, however, I am still listed as Unranked. Also, I do not appear on the SEOmoz Top Members List next to other users with 50 points. I'm guessing this is something that is updated once a day or so, and the rank will update tomorrow. Is my suspicion correct?
Moz Pro | | AnthonyMangia0