Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Is the update site crawl feature following robot.txt rules?

Moz Bar

593

jamestown Subscriber last edited by

I noticed that most of the errors would not be occurring if Moz's tool followed the rules implemented in sites robots.txt. Has anyone else seen this problem and do you know if Moz will fix this?
1 Reply Last reply
Reply Quote 0
Martijn_Scheijbeler @jamestown last edited by

I'm not a 100% sure about it, but probably in specific cases you want to have your own statements for Rogerbot.
1 Reply Last reply
Reply Quote 1
jamestown Subscriber last edited by

No because I thought Moz followed GoogleBot rules. Is this not the case?
1 Reply Last reply
Reply Quote 0
Martijn_Scheijbeler last edited by

Hi James.

Do you also have rules for the crawler of Moz in there? You can learn more about Roger here: https://moz.com/help/guides/moz-procedures/what-is-rogerbot

Martijn.
1 Reply Last reply
Reply Quote 2

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Moz not updating the spam score metrics

Hi Experts, I have done everything to come over the following metrics flagged as spam on my website by Moz, few months ago. But, Moz has not updated the spam score yet. ✓Low MozTrust or MozRank Score - improved from 2 to 4.5
✓Large Site with Few Links - My website have more than 6K backlinks
✓Small Proportion of Branded Links - My website have a good amount of branded backlinks
✓Thin Content - Every webpage on website has more than 500 words content
✓External Links in Navigation - There is no external link in navigation other than social media links
✓No Contact Info - The proper address has alreay been placed on website footer
✓Low Number of Pages Found - I am wondering if there are any standard score or number of links to reach to remove these flag?
Also, please suggest some ways that will help me improve moz spam score at faster rate. Thanks for your help in advance!"
Moz Bar | | jamesh.rich01

0
Crawling Problem : No-data Of My Site

Hey, Folks!! So It's Been More than a Month.And Moz is not crawling my website http://www.trickypedia.com/.I have seen Moz Last update I thought they will crawl my site but No results Disappointing. My Sites Domain Authority and Page Authority is still 1. But In other Seo Tools, they are Perfectly Crawling My Website. What Would Be the reason? Can anyone Please Explain.
Moz Bar | | seothatworks01

0
Why is Moz Crawling More Pages Than My Site Actually Has?

Hi I have a site that only has 5k pages but Moz has crawled 50K pages on the site when I initiated the site crawl. I don't exactly know why Moz is reporting me back so many pages but I was wondering why this is and if any of you out in the Moz community know anything about this. Thanks
Moz Bar | | drewstorys

0
Possible bug in Crawl Issues report?

Hi all - My crawl issues report shows 3 pages with missing titles. These are just google verification files and the robot.txt file - shouldn't these be excluded? Pages with Title Missing or Emptyas of May 11
URL Page Authority Linking Root Domains
https://www.mysite.com/googlea87e28121c071983.html
1 0
https://www.mysite.com/robots.txt
1 0
https://www.mysite.com/google9b9dc57478f61677.html
Moz Bar | | A-Drive

0
Verified News Sources: A New Feature in Fresh Web Explorer Launched Today!

Hey gang - I wanted to mention a small update we've made to Fresh Web Explorer (I was the architect on this one). It enables you to filter your searches and/or your Alerts to show only those sources/domains we've observed in Google News. E.g. here's a search for GigaOm with news filtering on (180 results in the last 7 days) vs. the same search with news filtering off (1,811 results). It's mostly useful for folks who are getting overwhelmed by the quantity of mentions they see in FWE and Alerts (I was experiencing that myself with mentions for my name and Moz) and just want to filter to the most important/noteworthy stuff. It's not perfect - Google News does have some weird domains that show in there occasionally - but it's definitely a much better filter if you have a bigger brand or term/phrase you're watching. Any questions or feedback, please let us know!
Moz Bar | | randfish

3
WP 4.0 Update Causing Major Duplicate Content Errors?

According to my moz analytics, my site has went through the roof with duplicate content. There's a nice Mozzer named Abe looking into this with me, but I'm wondering if it could be due to the WP 4.0 update. Has anyone else experienced an uptick like this before? I've never had any problems with the other updates. Thanks, Ruben
Moz Bar | | KempRugeLawGroup

0
Moz Crawl Test: Referrer is sitemap.gz?

Hi,
I'm looking at a crawl test report, and I'm seeing that most of the pages have the sitemamp.gz file listed as the referrer. As I recall in my other reports the referrer is usually the root domain - unless of course there's a redirect. Does having sitemap.gz as the referrer indicate a problem? If so, what problem does it indicate? Thanks!
Moz Bar | | Titan552

0
For some reason MOZ is not indexing anything but the home page of my site. Google appears to be indexing the whole site. But it makes my MOZ reports useless. Any recommendations?

MOZ only indexing home page of site.
Moz Bar | | NCCompLawyer

0