SEOMoz site crawlers created an issue for our servers
-
I have set up a number of campaigns with your pro tool. Unfortunately we have 7 sites on our server and our IT dept have said that we had an issue when your site crawlers visited for several sites at the same time - is there any way that I can retain the campaigns but have the sites crawled on request rather than automatically?
-
Hi Stephen,
If your server can't handle the SEOMoz spider, it wouldn't probably survive to all the malicious spiders out there... You should realy think into getting a new hosting account/server (or a new IT dept...).
In the "Campaign Settings" you have a "Weekly crawl completed _for campaign: _" checkbox that you could uncheck to disable the crawling of a certain campaign.
You might also want to use the "Crawl-delay:" directive into your robots.txt file, I don't know if it is supported by SEOMoz spider, but it might protect your server from other spiders.
Otherwise, like Barry Smith suggested, you should get in touch with SEOMoz team and see what they can do. If their crawler doesn't support the "Crawl-delay:" directive, you might want to suggest them to implement it.
Best regards,
Guillaume Voyer. -
Best to get in touch with them directly - http://www.seomoz.org/about/contact
customerservice@seomoz.org should get your email to the right people (as I don't know who best to contact about tools)
Alternatively you could use the helpdesk - http://seomoz.zendesk.com/home
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz can't crawl my site
Moz is being blocked from crawling the following site - https://www.cleanchain.com. When looking at Robot.txt, the following is disallowing access but don't know whether this is preventing Moz from crawling too? User-agent: *
Moz Pro | | danhart2020
Disallow: /adeci/
Disallow: /core/
Disallow: /connectors/
Disallow: /assets/components/ Could something else be preventing the crawl?0 -
Meta no index crawler warnings
I've decided that the duplicate content issues on my site weren't worth the effort from the amount of traffic the archive pages on my WordPress site received no I decided to no-index them using Yoast. Now I have 60 meta no-index crawler warnings. Should I just ignore these? It seems I get warnings, either way, I use the site. Does anyone have advice on how to move on with this?
Moz Pro | | Libra_Photographic0 -
Should I noindex user-created fundraising pages?
Hello Moz community! I work for a nonprofit where users are able to create their own fundraising pages on the website for supporters to directly donate. Some of them are rarely used, others get updated frequently by the host. There are likely a ton of these on our site. Moz crawl says we have ~54K pages, and when I do a "site:[url]" search on Google, 90% of the first 100 results are fundraising pages. These are not controlled by our staff members, but I'm wondering if meta noindexing these pages could have a big effect on our SEO rankings. Has anyone tried anything similar or know if this strategy could have legs for our site? My only concern is whether users wouldn't be able to find their fundraising page in our Google CSE implemented on the website. Any insight you fine folks could provide would be greatly appreciated!
Moz Pro | | Adam_Paris1 -
Creating a campaign and connecting it to Google Analytics
I want to create three campaigns in Moz and connect them to my Google Analytics account. However, two of the domains I am trying to create campaigns for are joined in Google Analytics. Do I need to separate the two in Google in order to create separate campaigns for each in Moz?
Moz Pro | | ERKSPAL0 -
How to set the crawler or reports to ignore
I have a mobile version of a site with a URL string that disables the mobile view on smartphones (view full site), string is like this example.com/page-name.html?mobile=off I need the seomoz pro reports or crawler to ignore it because the crawler visits both versions of the site then reports them as duplicate content. Is there a setting page I haven't visited yet that will set this?
Moz Pro | | Str82u0 -
Does SEOmoz give a way to know what link on what page produces the 404 errors that SEOmoz is telling me I have??
SEOmoz gives me a report of 404 errors on my site. Do they give a way to know from what link on what page produces the error?
Moz Pro | | MeridianGroup0 -
Any tools for scraping blogroll URLs from sites?
This question is entirely in the whitehat realm... Let's say you've encountered a great blog - with a strong blogroll of 40 sites. The 40-site blogroll is interesting to you for any number of reasons, from link building targets to simply subscribing in your feedreader. Right now, it's tedious to extract the URLs from the site. There are some "save all links" tools, but they are also messy. Are there any good tools that will a) allow you to grab the blogroll (only) of any site into a list of URLs (yeah, ok, it might not be perfect since some sites call it "sites I like" etc.) b) same, but export as OPML so you can subscribe. Thanks! Scott
Moz Pro | | scottclark0