Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do you stop Moz crawling a page?
-
Hello,
I have a contact form which generates thousands of duplicate crawl errors. I'm going to use to block Google indexing these pages. Will this also block MOZ from crawling these pages and displaying the error?
Thanks!
-
this is all well and good and I am able to do these, but how do I keep Moz from crawling an index.php file. our site is http://4signs.com no index file there at all so I'm not sure why it would be crawled.
thoughts?
-
Hi guys,
Awesome discussion so far
Yes, Chris is correct in that using noindex as a way to block Moz is not a effective way to do it. Since our tool is not a typical indexer (such as Google), we don't have some of the behavior of a normal spider. Instead, Roger is very good at rooting out issues that other crawlers might not notice. One thing Roger is also good at is obeying robots.txt.... you know him being a robot and all 
You can find more information about our friend here:
http://moz.com/learn/seo/robotstxt http://moz.com/help/pro/rogerbot-crawler
So if you are looking to block it from looking at a page without making content changes to your code, I would definitely look into using robots.txt. You can even use a user-agent specific directive to make sure you don't end up telling other robots/spiders to do the same thing.
I hope that helps! Please let us know if you have questions
Peter
Moz Help Team. -
On http://moz.com/help/pro/rogerbot-crawler Moz gives an answer to the question "We are still seeing duplicate content on Moz even though we have marked those pages as "noindex, follow. Any idea why?
Moz is not a search engine index, it uses a crawler. If those pages are not blocked by the robots.txt file, then Moz will crawl them. They ignore the noindex tag because they don't index anything. Search engines will honor the noindex tag and not index a page if you specify with the robots meta tag. However, to remove pages from the crawl, disallow them in the robots.txt or metarobots.
Their answer is not exactly clear, but according to it, no, a meta noindex will not block rogerbot from crawling your page.
-
Hi Gary,
You may find the following link helpful - http://moz.com/learn/seo/robotstxt on top of this you can read how to stop the moz bot here - http://moz.com/help/pro/rogerbot-crawler
If you have blocked bots from your page this will include the Mozbot. Hope this helps.
-
Yes it will.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is Moz more accurate than Ahrefs ?
When I check my this website https://joinpakarmy.com.pk/ on Moz it gives me DA 3
Moz Bar | | joinpakarmy234
But when I check it out on Ahrefs, it tells me there is no authority for this site.
So who is more accurate in the result?
thanks0 -
Moz Keyword Tool Monthly Volume
Ive recently put together a Keyword List of about 100 keywords on the Moz Keyword Explorer tool. One keyword, aerial filming, stood out as very low search volume of 51 - 100. I took the same 100 keywords and passed them through the Google Keyword Planner by Google AdWords. Aerial Filming has an average search volume of 1k - 10k according to the Keyword Planner. Even though Keyword Planner gives me a range of 1k - 10k, the lowest number is still 10 times higher than what the Moz Keyword Explorer was indicating. This drastic difference of volume was consistent across all 100 keywords. All of the Monthly Volume numbers were divided by 10. Why does Moz Keyword Tool display a search volume that is 10x less than what Google Keyword Planner is suggesting?
Moz Bar | | fictionarts0 -
Moz crawler only crawls one page?!
Hello there, I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled. We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working. Hopefully someone can point us at the right direction. Thanks in advance, Jeremy
Moz Bar | | mediaxplain.nl0 -
Has using Moz got me banned from Google search?
Dear fellow Mozzers, For the last couple of weeks when ever I do a google search I have to enter a captcha code. Now I hardly use google anymore which is annoying ! Then I went onto support.google.com and saw this: https://support.google.com/websearch/answer/86640?hl=en&ref_topic=3378866 "Automated traffic includes: Sending searches from a robot, computer program, automated service, or search scraper Using software that sends searches to Google to see how a website or webpage ranks on Google" Is Moz alone doing this? Thanks in advance for your response. Ash
Moz Bar | | -Ash-0 -
Duplicate Page and Title Issues
On the last crawl, we received errors for duplicate page titles and some duplicate content pages. Here is the issue: We went through our page titles that were marked as duplicate and changed them to make sure their titles were different. However, we just received a new crawl this week and it is saying there are even more duplicate page title errors detected than before. We're wondering if this is a problem with just us or if it has been happening to other Moz users. As for the duplicate content pages, what is the best way to approach this and see what content is being looked at as a "duplicate" set?
Moz Bar | | Essential-Pest0 -
Getting 'Sorry, but that URL is inaccessible' error msg when trying to run On-Page Grader
I just signed up for MOZ Pro for the first time today. Tried to run the 'on-page grader' tool on some of my pages but I'm getting a 'Sorry, but that URL is inaccessible' error msg. I have verified against the robot.txt file that the pages are NOT blocking any crawlers. Can anybody help?
Moz Bar | | spinoki0 -
Weird back link showed in moz crawl
Some time ago somebody from this site: http://dianibeach.com created a weird link to our site which had on the end db. Later we have realized that the link was coming from every footer on each page. I believe that the back links from footer does not have realy value and even the more of them the less value. We have asked the guy to remove that links as I thought it might harm our site more then help. Now I I was very surprised to find this link in moz crawl error as second top page on our site in current index??? Can somebody explain how is this possible?? The most ridiculous thing is that when I click on that link it realy opens our site! How is that possible, what is it? This is the link: http://villasdiani.com/?db Thank you very much for any help with this
Moz Bar | | Rebeca10 -
Is Moz Analytics still available to all MozCon attendees?
I thought all Mozcon attendees were going to get access to Moz Analytics even after the convention was over. I can't seem to find were to access it from my campaigns manager dashboard and the Analytics landing page says im on queue to get access.
Moz Bar | | AmberHanson0