How do you stop Moz crawling a page?
-
Hello,
I have a contact form which generates thousands of duplicate crawl errors. I'm going to use to block Google indexing these pages. Will this also block MOZ from crawling these pages and displaying the error?
Thanks!
-
this is all well and good and I am able to do these, but how do I keep Moz from crawling an index.php file. our site is http://4signs.com no index file there at all so I'm not sure why it would be crawled.
thoughts?
-
Hi guys,
Awesome discussion so far
Yes, Chris is correct in that using noindex as a way to block Moz is not a effective way to do it. Since our tool is not a typical indexer (such as Google), we don't have some of the behavior of a normal spider. Instead, Roger is very good at rooting out issues that other crawlers might not notice. One thing Roger is also good at is obeying robots.txt.... you know him being a robot and all 
You can find more information about our friend here:
http://moz.com/learn/seo/robotstxt http://moz.com/help/pro/rogerbot-crawler
So if you are looking to block it from looking at a page without making content changes to your code, I would definitely look into using robots.txt. You can even use a user-agent specific directive to make sure you don't end up telling other robots/spiders to do the same thing.
I hope that helps! Please let us know if you have questions
Peter
Moz Help Team. -
On http://moz.com/help/pro/rogerbot-crawler Moz gives an answer to the question "We are still seeing duplicate content on Moz even though we have marked those pages as "noindex, follow. Any idea why?
Moz is not a search engine index, it uses a crawler. If those pages are not blocked by the robots.txt file, then Moz will crawl them. They ignore the noindex tag because they don't index anything. Search engines will honor the noindex tag and not index a page if you specify with the robots meta tag. However, to remove pages from the crawl, disallow them in the robots.txt or metarobots.
Their answer is not exactly clear, but according to it, no, a meta noindex will not block rogerbot from crawling your page.
-
Hi Gary,
You may find the following link helpful - http://moz.com/learn/seo/robotstxt on top of this you can read how to stop the moz bot here - http://moz.com/help/pro/rogerbot-crawler
If you have blocked bots from your page this will include the Mozbot. Hope this helps.
-
Yes it will.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Links and images in custom css are not recognised by Moz Toolbar or Yoast plugin
Hi, I am building a website in Wordpress and when checking the pages through Moz toolbar or Yoast plugin, I notice that these don't recognise the internal links in custom CSS and the images neither. Here some code to clarify how it is built: [vc_row][vc_column width="1/2"][ultimate_heading main_heading="Veldrijden" alignment="left" main_heading_margin="margin-bottom:20px;"][/ultimate_heading][vc_single_image image="6497" img_size="full" onclick="custom_link" link="http://k488239.hosting.kinamo.be/activiteiten/veldrijden/"][vc_btn title="Meer info" color="mulled-wine" align="left" link="url:http%3A%2F%2Fk488239.hosting.kinamo.be%2Factiviteiten%2Fveldrijden%2F|||"][/vc_column] The links are embedded in images and in 'More info' buttons. Also the internal links in the main website navigation are not recognised, but the normal a href links are recognised. For the images it is a little different: the alt text from the images gets recognised by Moz, but Yoast says there are no images on the page. The only thing I can think of that there is no 'a href' in the links and for the images that it has something to do with the page builder. Can this effect my SEO results? Thanks.
Moz Bar | | Mat_C0 -
Current Title is short but Moz show error that Title is Too Long
Hello, The current title of an article is of 30 characters. But in Moz it say "Title Element Too Long". The URL of the article is long. Please let me know why Moz is showing it as "Title Element Too Long"
Moz Bar | | ProcessSEO0 -
Different Errors Running 2 Crawls on Effectively the Same Setup
Our developers are moving away from utilising robots.txt files due to security risks, so e have been in the process of removing them from sites. However we, and our clients still want to run Moz crawl reports as they can highlight useful information. The two sites in question sit on the same server with the same settings (in fact running on the same Magento install). We do not have a robots.txt files present (they 404), and as per Chiaryn's response here https://moz.com/community/q/without-robots-txt-no-crawling this should work fine? However for www.iconiclights.co.uk we got: 902 : Network errors prevented crawler from contacting server for page. While for www.valuelights.co.uk we got: 612 : Page banned by error response for robots.txt. These crawls were both run recently, and there was no robots.txt present. Not to mention, they are on the same setup/server etc as mentioned. Now, we have just tested this, by uploading a blank robots.txt file to see if it changed anything - but we get exactly the same errors. I have had a look, but can't find anything that really matches this on here - help would really be appreciated! Thanks!
Moz Bar | | I-COM0 -
MOZ crawl test is not reporting on all the pages on my site.
I've run the crawl test one of the sites I've taken over SEO for, however its only picking all the pages. For instance it indexes all the pages under xxxxx/us but none under xxxxx/au or xxxxx/uk The pages are being indexed as they're ranking in Google. Thanks.
Moz Bar | | ahyde0 -
On Page Grader Changed?
Hi, Could anyone tell me if the on page grader has changed? I remember seeing 8 boxes at the top but now i'm only seeing 6... Before I remember seeing a H2-H4 and a Strong/B Box? Thanks
Moz Bar | | dmxsta0 -
SEO MOZ ERROR
Hello moz comunity, I tried to use the moz keyword difficulty service in the last 2 days and I get this error over and over again... see photo: http://www.evernote.com/shard/s238/sh/5775a179-1be7-4e76-8563-cf087c37cf2b/576bda1a72f446a8806a0f1914193829 Oops Gosh! It looks like something has gone a bit wrong. Don't worry though, we know and are fixing it. How Can I solve this? I need to check a lot of keywords for my websites. Any alternatives? Thank you !!!
Moz Bar | | Sebastyan220 -
On the on page optimization page, I found out that there are 2 contributing factors which are opposite to each other. "No More Than One H1 Tag" and "Appropriate Keyword Usage in H1 Tag"
"No More Than One H1 Tag" and "Appropriate Keyword Usage in H1 Tag" If you fulfill one condition, the other one is not completed. If you consider Article heading as H1 then Moz do not detect keyword in the heading.
Moz Bar | | MoeezLodhi0
