Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Is the update site crawl feature following robot.txt rules?

Moz Bar

593

jamestown Subscriber last edited by

I noticed that most of the errors would not be occurring if Moz's tool followed the rules implemented in sites robots.txt. Has anyone else seen this problem and do you know if Moz will fix this?
1 Reply Last reply
Reply Quote 0
Martijn_Scheijbeler @jamestown last edited by

I'm not a 100% sure about it, but probably in specific cases you want to have your own statements for Rogerbot.
1 Reply Last reply
Reply Quote 1
jamestown Subscriber last edited by

No because I thought Moz followed GoogleBot rules. Is this not the case?
1 Reply Last reply
Reply Quote 0
Martijn_Scheijbeler last edited by

Hi James.

Do you also have rules for the crawler of Moz in there? You can learn more about Roger here: https://moz.com/help/guides/moz-procedures/what-is-rogerbot

Martijn.
1 Reply Last reply
Reply Quote 2

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Is there a report or a place in Moz that I can see Hreflang errors for my site?

Is there a report or a place in Moz that I can see Hreflang errors for my site?
Moz Bar | | av-seo

1
How to find the pages in the site which is ranking for zero keyword

I have around three thousand pages in my website,how to find the list of pages which is ranking for zero keywords
Moz Bar | | srinivasan.n

1
Moz Crawler Causing Server Timeouts... Crawling thousands of non-existant pages with query parameters

Moz crawler is crawling all pages like this: http://www.xxxx.com/?product_count=100&product_order=desc&product_orderby=date http://www.xxxx.com/?product_count=100&product_order=desc&paged=1 http://www.xxx.com/?product_count=100&product_order=desc&product_view=grid Last month it crawled 80,000 pages on a site with less than 100 pages. Is there a way to select only certain pages to be crawled? Right now it is still crawling this site, since Monday morning and it's Tuesday mid-day. Every Monday it is causing time-outs from high band width on our server. Just getting ready to delete this client from the account unless there is a solution someone can give us. Thanks.
Moz Bar | | adirondack

0
Internal Links Count in Crawl Report

My understanding of the 'Internal Links' results in a moz crawl report is that it represents the number of links on the given page that link to other pages on the same site.Assuming this is a correct assumption: We recently ran a crawl report on www.phase1tech.com. Some of the pages are coming back with a large amount of 'internal links'. These 2 pages for example are showing 800 internal links: http://www.phase1tech.com/Upcoming-Events
http://www.phase1tech.com/Contact Then there are a number of pages coming back with 705 Internal Links, including: http://www.phase1tech.com/Dalsa-CameraLink-Cameras
http://www.phase1tech.com/Hitachi-CameraLink-Cameras At best there are approximately 70-80 links on these pages. Where are these large counts coming from? Is there a means to see what the links being reported on are? At the same time the 'Too Many On-Page Links' indicates 'No' for some pages with a high number of links, and 'Yes' for pages with a low number of links. For example: http://www.phase1tech.com/Baumer-SX-Series
Too Many On-Page Links: Yes
Internal Links: 2
What's up with that?
Moz Bar | | AISEO

0
WP 4.0 Update Causing Major Duplicate Content Errors?

According to my moz analytics, my site has went through the roof with duplicate content. There's a nice Mozzer named Abe looking into this with me, but I'm wondering if it could be due to the WP 4.0 update. Has anyone else experienced an uptick like this before? I've never had any problems with the other updates. Thanks, Ruben
Moz Bar | | KempRugeLawGroup

0
Is there a way to get Page Authority values included in the Crawl Diagnostic .csv export?

Would be nice to have these values included so that you can sort by Page Authority. 4uF6efx.png
Moz Bar | | WebReputationBuilders

0
Site Crawler Tool by the Company Formerly Known As SEOMoz

Moz had a tool I used that would crawl my site and send me a report of all pages, all errors, 301s 404s 505s, and a whole plethora of stuff. I used it to fix pesky errors quite a bit. Does this still exist? Was it replaced or am I just not finding it in the new design?
Moz Bar | | KJ-Rodgers

0
Not getting foreign characters in crawl diagnostics .csv

The crawl diagnostics .csv file is showing high-ascii characters instead of the correct language (foreign language website) e.g. Vietnamese, Chinese (both kinds), etc. Is there a way to get this right?
Moz Bar | | trainSEM

0