Why does SEOMoz crawler ignore robots.txt?
-
The SEOMoz crawler ignores robots.txt
It also "indexes" pages marked as noindex.
That means it is filling up the reports with things that don't matter.
Is there any way to stop it doing that?
-
Hi Alan,
The code should be ok
Try to "drive-test" it with a custom crawl from http://pro.seomoz.org/tools/crawl-test then you will see if it works well.
I am glad the link was useful.
Gr.,
Istvan
-
Thank you István
I added this:
User-agent: rogerbot
Disallow: /sendtoafriend/
Disallow: /photo/
Disallow: /pix/- because crawlers shouldn't go down those paths and roger is detecting pages without descriptions.
Is what I added OK?
-
Hi,
You can block RogerBot from Robots.txt
Check for further instructions on: http://www.seomoz.org/dp/rogerbot
"Please note: Adding this code will prevent our crawl test tool from being able to crawl your website."
Gr.,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I exclude a sub-domain from SEOMoz campaigns?
We have recently implemented a white label site that is on a sub-domain. The site employs noindex on most of the pages I imagine due to duplicate content concerns on other white label versions of the site. It has led to a spike of over 14 thousand notices on our report. Is there a way to exclude a sub-domain from the SEOMoz scans and reports?
Moz Pro | | TSDigital0 -
What software can I use on my Mac to open and read a SEOMoz CSV exported file?
I do not want to buy XL or Pages just to read the CSV from SEOMoz. So I bought an app on the AppStore... and this app is unable to read the CSV from SEOMoz. Since I already wasted $2, Id rather avoid to waste more (and avoid that to others too!). What software is recomanded to open these CSV files? Also, I tried Google Docs, but I bumped in their 400K cells limit 😞
Moz Pro | | jgenesto0 -
Why not having on SEOmoz pro dashboard the table shows by rand fishkin in his whiteboard friday (see below)
I have seen this video, very interesting. Why not building this keyword dashboard in SEOmoz as all data can be taken from Google Analytics ? http://www.seomoz.org/blog/keyword-metrics-for-seo-and-driving-actions-from-data-whiteboard-friday
Moz Pro | | betadvisor0 -
SEOMoz SERP Overlay Vs. Opensite Explorer
This may be a simple answer, but when I search for a term and see a website on the SERP, the SEOMoz tool bar shows link data beneath the link. For this site in particular, it shows 300+ links for the root domain and for the page, it shows 8 links. When I put that website in opensite explorer, it shows the same data for the "Page" stats (8 links from 3 root domains), but nowhere can I find where the 315 links from 4 domains comes from. So my questions is why do those links show in the SERP overlay, but not in OSE? il5y9.png
Moz Pro | | GetFoundFirst0 -
Why Is SEOMOZ No Longer crawling All Of My Site
Hi all, I joined Seomoz over a month ago and Roger has been crawling all of the pages on the site approx 20 pages. Through out the last few weeks I have been working on the errors and notices identified by Roger. However, this week Roger has only re-crawled 1 page and is not picking up all the other pages. Has any one come across this problem. can you recommend any thing to resolve it? Many thanks in advance....
Moz Pro | | Dan280 -
Why is the SEOmoz crawler crawling the old version of our website?
Hello, I'm a new SEOmoz member. On Dec. 2nd, after completely redesigning our website, we migrated to a new hosting company by switching our DNS to the new server. The vast majority of the URLs have changed and we configured redirects of the old URLs to the new ones. Although, this task is not completed yet. After the migration, I created an account on SEOmoz to be able to track our progress and find the issues to fix to optimize our SEO. For some reason, in the SEOmoz reports it is the old URLs that show up. Unless the crawler does not actually crawl the pages and only uses the indexed pages to generate its report, I don't understand how could this possible. Anyone has a clue? When will the new URLs be indexed by SEOmoz and the major search engines? Thanks for your help!
Moz Pro | | Gestisoft-Qc0 -
Why is Roger crawling pages that are disallowed in my robots.txt file?
I have specified the following in my robots.txt file: Disallow: /catalog/product_compare/ Yet Roger is crawling these pages = 1,357 errors. Is this a bug or am I missing something in my robots.txt file? Here's one of the URLs that Roger pulled: <colgroup><col width="312"></colgroup>
Moz Pro | | MeltButterySpread
| example.com/catalog/product_compare/add/product/19241/uenc/aHR0cDovL2ZyZXNocHJvZHVjZWNsb3RoZXMuY29tL3RvcHMvYWxsLXRvcHM_cD02/ Please let me know if my problem is in robots.txt or if Roger spaced this one. Thanks! |0 -
SEOMoz PR vs. Google PR
I have the SEO MozBar set up and for one of the sites we are maintaining, it is showing them on a 59 as PA in the MozBar. When I check it on Google PR, it has either no info or a 0. http://www.prchecker.info was one of the ones I found which shows them as 0. Any help would be greatly appreciated.
Moz Pro | | Champions0