Can I prevent some pages from being crawled from SEOMoz spider and still not affect Google Spider?
-
Well, basically that's the question
Can I prevent some pages from being crawled from SEOMoz spider and still not affect Google Spider?This is, I have more than 10.000 pages on the website, and I am not interested in having reports for many of them, but I still wanna get SEO visits on them, so I want Google to crawl it easily...
Thanks!
-
Hey Martijn,
Thanks!
What about if I want to avoid rogerbot from crawling a range of pages from let's say /000001/; /000002/... /200000/ ?
Thanks!
-
Hey Martijn,
Thanks!
What about if I want to avoid rogerbot from crawling a range of pages from let's say /000001/; /000002/... /200000/ ?
Thanks!
-
Hi Matt,
Absolutely, you could do this by adding a special part to your robots.txt file specified to the user agent rogerbot which is used by SEOMoz for their spider. This could be done with for example:
User-agent: rogerbot Disallow: */anythingyouwanttoexcludeforroger/*Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Find 28 pages (status code 200) on MOZ Pro but 8 on Screaming Frog SEO Spider 11.3
Hi, Does anybody have an idea about this please? I am a newbie, so I am must be doing something wrong. The website: vtcservice-plus.com Thanks
Moz Pro | | ethan30000 -
Why might Google be crawling via old sitemap, when the new one has been submitted and verified?
We have recently relaunched Scoutzie.com and re-submitted our new sitemap to Google. When I look on Webmaster tools, our new sitemap has been submitted just fine, but at the same time, Google is finding a lot of 404s when crawling the site. My understanding, it is still using crawling the old links, which do not exists. How can I tell Google to refresh it's index and to stop looking at all the old links?
Moz Pro | | scoutzie0 -
Where has the old seomoz crawl tool gone? I can't seem to find it
I'm looking for the (SEO)moz crawl tool - but can't find it. Where has it gone?
Moz Pro | | SearchMotion0 -
Does SEOmoz give a way to know what link on what page produces the 404 errors that SEOmoz is telling me I have??
SEOmoz gives me a report of 404 errors on my site. Do they give a way to know from what link on what page produces the error?
Moz Pro | | MeridianGroup0 -
Hyphens in Page Titles?
We are using a combination of keywords using our brand name. So the keyword is structure as: brand name - word (separated by a hyphen) When I run a report on the page for the keywords that have the above format, the report tells me that I need to use the keyword in the title of the page. Is it okay to have hyphens in Page Titles? I assume not, but I want to double check. Thanks, Alex
Moz Pro | | costarica.com0 -
Issue in number of pages crawled
i wanted to figure out how our friend Roger Bot works. On the first crawl of one of my large sites, the number of pages crawled stopped at 10000 (due to the restriction on the pro account). However after a few weeks, the number of pages crawled went down to about 5500. This number seemed to be a more accurate count of the pages on our site. Today, it seems that Roger Bot has completed another crawl and the number is up to 10000 again. I know there has been no downtime on our site, and the items that we fixed on our site did not reduce or increase the number of pages we had. Just making sure there are no known issues with Roger Bot before I look deeper into our site to see if there is an issue. Thanks!
Moz Pro | | cchhita0 -
Crawl Rate for Lower Page Authority Websites
Hi,At thumbtack.com we get tons of links from low (or no) page authority websites, and I'm wondering what the crawl rate of those links looks like. I know Google pulls in the web at an astonishing rate, but I'd imagine they aren't re-crawling lower PA very frequently.Are they discovering these links a week after they're posted? A month? More? I spent a while looking around for histograms of actual crawl rates and found surprisingly little. I'd love to see average crawl rate by Domain or Page Authority if that exists anywhere.
Moz Pro | | Thumbtack
Thanks!-MichaelP.S. Here are some random examples of the types of pages with inbound links I'm talking about. Normally we wouldn't spend too much time thinking about these, but there's just so many of them we can't ignore it!- http://www.majestic-cleaners.webs.com/- http://domchieraphotography.blogspot.com/- http://charlottepiano.musicteachershelper.com/- http://pin-upgirlphotography.vpweb.com/default.html- http://jfaithful.weebly.com/0 -
Where in SEOMoz pro can I determine how many .edu and .gov links a competitor's site has?
Like in the SEOBook toolbar where you can click the information button and get the number of links to the domain and specific page for .edu and .gov sites. Thanks, Daniel.
Moz Pro | | iSenseWebSolutions0