Can I prevent some pages from being crawled from SEOMoz spider and still not affect Google Spider?
-
Well, basically that's the question Can I prevent some pages from being crawled from SEOMoz spider and still not affect Google Spider?
This is, I have more than 10.000 pages on the website, and I am not interested in having reports for many of them, but I still wanna get SEO visits on them, so I want Google to crawl it easily...
Thanks!
-
Hey Martijn,
Thanks!
What about if I want to avoid rogerbot from crawling a range of pages from let's say /000001/; /000002/... /200000/ ?
Thanks!
-
Hey Martijn,
Thanks!
What about if I want to avoid rogerbot from crawling a range of pages from let's say /000001/; /000002/... /200000/ ?
Thanks!
-
Hi Matt,
Absolutely, you could do this by adding a special part to your robots.txt file specified to the user agent rogerbot which is used by SEOMoz for their spider. This could be done with for example:
User-agent: rogerbot Disallow: */anythingyouwanttoexcludeforroger/*
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Problem with On-page
I have an issue. I have added 5 keywords but when i go to the "on page" tab. They are not there... So i press on "Add keyword" and it takes me to another page where i can see all my keywords. So i go back to the "on page" and no keyword shows up. I wanna have a summary of the weekly crawl for the on page of these keywords and it's not showing up 😞 Anybody knows why?
Moz Pro | | theseolab0 -
Can I see when SEO Moz has crawled my website?
I would like to know if it's possible to see (maybe in my Google Analytics) if SEO Moz has crawled my website. I'm also curious if and where I can see when the robot of Google visited my website. Thanks!
Moz Pro | | Spotler0 -
Why does Seomoz report other values then Google Webmaster?
Dear Seomoz Community I check my Seomoz ranking from time to and today I saw a big discrepancy between the reported domains linking to my site in the Site Explorer and Google Webmaster tools. Seomoz reports 19 linking domains
Moz Pro | | menelik
Google shows 150 domains linking to my site. Is there an explanation for this discrepancy? Thank for helping me out 😉0 -
Why are the rankings of my keywords different to rankings from other programmes such as 'Access my Lan' and 'Firefly'? Does SEOMoz count 'Google Places and Maps' in their rank count? Im thinking this could explain the difference
Why are the rankings of my keywords different to rankings from other programmes such as 'Access my Lan' and 'Firefly'? Does SEOMoz count 'Google Places and Maps' in their rank count? Im thinking this could explain the difference
Moz Pro | | john_Digino0 -
Recent SEOMoz Crawl = Strange Results
Did anyone else get some really strange results in their weekly crawls this week with the campaign tool? Either my ranks sky rocked across three different sites or the tools is busted. Something to the tune of having 4 pages ranking in the top 30 to now having 15-16 pages ranking in the top 30. I'd love to find out it is just all the hard work paying off but i am worried it is the later. Regards - Kyle
Moz Pro | | kchandler0 -
After fixing errors can I re-crawl for diagnostics?
As I am fixing errors will the campaign automatically update to show where I have fixed issues?
Moz Pro | | eidna220 -
Where in SEOMoz pro can I determine how many .edu and .gov links a competitor's site has?
Like in the SEOBook toolbar where you can click the information button and get the number of links to the domain and specific page for .edu and .gov sites. Thanks, Daniel.
Moz Pro | | iSenseWebSolutions0 -
Crawl Issues
My website - qtmoving.com - has 26 articles and when the SEOmoz did a crawl it only found 13 articles. Can someone please give me some insight as to why not all pages are being crawled.
Moz Pro | | CohesiveMarketing0