Welcome to the Q&A Forum

Generally speaking, CAPTCHA based DDoS protection is not advised as it will drive away clients and DDoS bots alike.
Same goes for delay pages.

Also, DDoS attacks are swift and can bring site/server down in minutes so the manual approach is risky and probably less effective than it sounds on paper. Even if you are always near your PC, like many of us here are, you`ll still be exposed on weekends, nighttime, launch breaks, etc... Hackers know that and will exploit it. (favorite attack time - VERY EARLY Sunday mornings... )

You should also consider that DDoS can serve as a means to an end. It's often used to "soften" the protective layer to hack the database, inject the site with backdoors, deface or delete it and so on and so forth.

What you really need is a combination of auto-triggering and transparent mitigation, which shies away from intrusive mitigation methods in favor of more subtle progressive challenges (i.e. JS challenge combined with reputation and behavior monitoring).
This may sound fancy but it really this is the emerging industry standard and growing competition between vendors actually drives costs down.

Hi
SEO wise, there are many things to consider before on-boarding a CDN service and bot filtering is one of them.
I've covered the topic in 2 blogs posts about a year ago:

1. CDN's SEO related advantages 2. CDN's SEO related myths (and possible issues to consider)

PS: I work for CDN-based security and acceleration service provider - Incapsula.

Hi Ian
These are all good suggestions. Accidentally (SSL excluded) these are all covered by modern CDNs, which do much more than just proxy.

• Memory vs. disk caching: Memory is faster

  CDNs will cache from memory by providing free/low-cost access to very large pool of resource, which most website owners couldn't utilize otherwise. (each proxy location will have ~10 high-powered servers that allocate memory for caching)

• Server configuration: Is your database on the same server as your web server? That's a problem, if true.

  This is a core CDN capability. The cached web content is served from proxy while database complies in the background and serves the rest of the materials.

• Database caching: Is your e-commerce system using it? The first hit on the database is a huge hit on performance.

  This is a great suggestion and - unlike other points - this is not a default CDN feature. However, this problem can be solved by intelligent caching heuristic. What I mean is that by monitoring resource usage over large pool of visitors an intelligent CDN system can identify dynamically generated objects which are not often changed, yet still un-cached due to their dynamic "origin".

  Pinpointing such objects and caching them in the way that ensures personalization and freshness will reduce the impact of database processing. For example, typical e-commerce site and will dynamically generate the product list from DB when in fact most products specifics (the image, the text, the pricing, etc) will not change over the product life-cycle. For all states and purposes these are static resources, yet they are being generated dynamically, for the lack of the better option...

  If you CDN can identify such instances and move in to cache these parameters, you will benefit from 30-50% improvement, on top of the usual 30-40% CDN factor.

  What I`m describing here is not future-tech but a patent-pending algorithms which are already used by the industry. You can find out more here:http://www.incapsula.com/the-incapsula-blog/item/414-advanced-caching-dynamic-through-learning

Honestly, I don't know. I don't think TTFB was ever comparatively tested - at least no to the best of my knowledge.

For security, these are some of the resources I can point to.
I understand that this is not the main issue
Still, I wanted to provide some factual context to my previous statements.
http://zeroscience.mk/files/wafreport2013.pdf http://ddos-protection-services-review.toptenreviews.com/ http://tonyonsecurity.com/2012/11/13/protecting-your-website-cloudflare-or-incapsula/ 
(This last one is interesting since Tony is a COO of Sucuri. Some would call his our competitor. I prefer 'colleague' )

Hi Vadim

Thanks.
Yep, I work for Incapsula but no, we are not the said "Mod".  

As for CF comparison... Generally speaking, we are more business oriented and security focused. I know that our security offering is more comprehensive, especially because both WAFs were comparatively pen-tested on several occasions and we always came out as consistently (and significantly) better option.We also have addition security features - like 2FA support and backdoor shell protection - which CF simply doesn't offer and we do more in way of ddos mitigation, especially against smart application layer attacks which require security capabilities, besides network muscle.

Still, speed wise, I always considered us to be pretty much on the same level. However, until few days ago I never considered TTFB to be such core SEO factor, so maybe we have better performance there...  
But again, to be fair, I`m only speculating - mostly based on the CF blog you've shared.
(if TTFB is considered un-important, it might also be under developed...)

Might be an interesting thing to test and document.

I absolutely agree with Vadim. (+1)

Google is the best source for Google facts. Everything else is just speculation.

And yes, generally speaking, the best answer is to use a CDN.... 
The reason is simple. CNDs proxy technology, which was designed to minimize "physical" distances between the site's content and browsers, directly influences TTFB.

Being an in-house SEO for a CDN company I get a lot of questions about this from our support and clients. I have to admit, until recent Moz post, I wasn't aware of full implications of TTFB and considered it to be one of few page load speed related metrics.  (http://moz.com/blog/how-website-speed-actually-impacts-search-ranking)

This post really helped me get a better grasp on things. Interestingly enough, few month ago one of our clients Guest Posted in our blog about speed improvement gained by our free plan. Among other things, he mentioned 70% improvement in TTFB (grade going from F to A) 
(http://www.incapsula.com/the-incapsula-blog/item/718-what-incapsula-free-did-for-my-site)

At the time I didn't give it much attention. Because, like many others, I was focusing on overall load speeds.... 
Now I can't help but feel that this was a missed opportunity.
This post could be even better with the added SEO angle...
If anyone here is interested in giving this a try and guest posting about it, I`ll be happy to provide all resources needed on our end.

Generally speaking, the best answer is to use a CDN.
CNDs proxy technology, which was designed to minimize "physical" distances between the site's content and browsers, directly influences TTFB. Being an in-house SEO for a CDN company I get a lot of questions about this from our support and clients. I have to admit, until recent Moz post, I wasn't aware of full implications of TTFB and considered it to be one of few page load speed related metrics.  (http://moz.com/blog/how-website-speed-actually-impacts-search-ranking)

This post really helped me get a better grasp on things. Interestingly enough, few month ago one of our clients Guest Posted in our blog about speed improvement gained by our free plan. Among other things, he mentioned 70% improvement in TTFB (grade going from F to A) 
(http://www.incapsula.com/the-incapsula-blog/item/718-what-incapsula-free-did-for-my-site)

At the time I didn't give it much attention. Because, like many others, I was focusing on overall load speeds....

I can't help but feel that this was a missed opportunity. This post could be even better with the added SEO angle...
If anyone here is interested in giving this a try and guest posting about it, I`ll be happy to provide all resources needed on our end.

What's the end goal here?
Are you actively trying  to block all bots?

If so, I would still suggest "Disallow:/".
The other syn-text may also work, but if Google suggests using a backslash, you should probably use it.

Hi Lisa

This is interesting.

As far as I can tell, everything is configured 100% correctly.

Few questions:

1. Do you keep all images in /sites/default/files/?
    Were all (most) of your errors related to items using that path?

2. This could be a downtime issue. Did you experience any availability problems?

3. Do you see any changes in the error report? (different results for the same URL)
    If so, this makes my downtime theory more plausible.

Also, the site loaded VERY slowly for me.

I've run Google Page speed and saw that you have issue with Cache headers (low expiration date that makes it harder to leverage local and CDN caching capabilities)

See here: https://developers.google.com/speed/pagespeed/insights#url=http_3A_2F_2Fwww.mnn.com_2F&mobile=false&rule=LeverageBrowserCaching

This should be looked at. Beyond the immediate ux and seo implications, this can contribute to server load and can be have a negative effect on website's availability. (i.e. the above mentioned downtime)

Best

Igal

Thanks.

100% agree with the Meta Noindex suggestion.

Didn't say it wasn't.

I`m just not sure how these rules apply to parameters, since they are not a part of the "core" URL.

(For example: What happens if I take a URL from your site, change a nocrawl=1 to nocrawl=0 and link to it from mine?
Do you have any URL sanitation rules in place to overcome that or will the page be indexed by Googlebot when it crawls my site and moves on to yours?)

Personally, when dealing with parameters, I find it easier to work with WMT so I was offering an easier workaround, (at least for me)

To tell you the truth, I would use hard-coded on page meta noindex/nofollow here (again, as parameters can be so easily manipulated).

Hi

I`m not sure if this is the best way to go about it.

Robots.txt is commonly used for folder level disallow rules, I`m not sure how it will respond to parameters.

Having said that, there are several things you can do here:

1. You can use WMT to zero in on this parameter and  prevent it from being searched.
To do so choose Configuration>>URL Parameters, answer "Yes" to the question about content change and  
    check-in the 3rd bullet (Only URL with value...) Of course you'll need to choose "1" as the right value.

2. If this still didn't solve your issue, you might want to try using htacess + regex to prevent access by user agent.
    You can find user-agent information here Googlebot user agent list

Also, you may want to check my blog post  about some of the less known Googlebot Facts (shameless self-promotion)

Best

Igal

Hi

This is a valid concert.

As Mat correctly stated, Googlebot is not easily manipulated.
Having said that, Googlebot impersonation is a sad fact.

Recently we released a Fake Googlebot study in which we've found out that 21% of all Googlebot visits are made by different impersonators - fairly "innocent" SEO tools used for competition check-ups, various spammer and even malicious scanner that will use Googlebot user-agent to try and slip in between the cracks and lay a path for a more serious attack to come (DDoS, IRA and etc).

To identify your visitor can use Botopedia's "IP check tool"  - it will cross-verify the IP and help reveal most fake bots. 
(I`ve already searched for 66.249.71.25 and it's legit - see attached image)

Still, IPs can be spoofed.
So, if in doubt, I would promote a "better safe than sorry" approach and advise you to look into free bad bot protection services (there are several good ones).

GL

TBHPK

Oleg gave a great answer.

Still I would add 2 things here:

1. Go to GWMT and under "Health" do a "Fetch as Googlebot" test.
     This will tell you what pages are reachable.

2. I`ve saw some occasions of server-level Googlebot blockage.
If your robots.txt is fine and your page contains no "no-index" tags, and yet you still getting an error message while fetching, you should get a hold on your access logs and check it for Googlebot user-agents to see if (and when) you were last visited.

This will help you pin-point the issue, when talking to your hosting provider (or 3rd party security vendor).

If unsure, you can find Googlebot information (user agent and IPs ) at Botopedia.org.

Putting a noindex/nofollow on an index url will remove it from SERPs, although some ulrs will still show for direct search (using the url itself as a KW) but even then they will appear as clear links without any TItle/Description details.

Using a 301 redirect will remove the old page from index, regardless of noindex/nofollow.

If you are using a noindex/nofollow for the new url - both will not show.

Good luck

(+1)  For redirect to main category page option. I did this several time, including for a very large tourism site which had a LOT of "inventory" changes (we are talking about dozens-hundreds/day) and had great results.

One thing I would like to suggest is to look into doing 302 and removing the redirects after 2-3 month.

The reason for this is purely practical. In our case, after just a few month, we were looking at many thousands of redirects and this is not something you want to "carry around".
My suggestion allows you to still make use of link juice for removed pages and, at the same time, have a manageable redirect profile.

As a safe net you can have a generic: "404 >>> 301 >>> Homepage" rule underneath.

Hi

1st of all (to address previous comment) Meta description is not an SEO factor.

"Even though we sometimes use the description meta tag for the snippets we show, we still don't use the description meta tag in our ranking."
Source: http://googlewebmastercentral.blogspot.fr/2009/09/google-does-not-use-keywords-meta-tag.html

Meta titles however are a major factor and duplicate meta titles might point to Canonicalization issue.

To solve, first investigate (scrape SERP or use Webmaster Tools to identify error source and see if you are dealing with duplicated Titles or Duplicated Content - > similar pages with the same Titles/Content that appear under different URLs)

If Duplicated Titles only, find a way to optimize all Tags on site (a must)

If Duplicated Content, use rel=canonical to solve your issue: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=139394

GL

Not necessarily.  Can you provide a link to Google SERP so I can answer this accurately?
Sometime (pretty often) Google will show one page but provide the rankings due to another page.

For example, I have a strong "CCTV Cameras" category page and under I have a "[Specific Type of CCTV Camera] " page. The first page is strong and he will provide me with high ranks but it dosen't mean it will be show for it. Instead, due to visitor data or/and social signals, the second page may appear if it's temporarily more popular.

After it gets removed (i.e. sold out) the first page, or other trendier topic related page, will assume the position.

This scenario can be discovered by looking at 100 results (use. advance search) there sometimes youll find several results from the same site, side by side - where in basic search youve saw only one single result. This is a symbiotic relationship I`m talking about. The 1st and displayed result is not the "cause" for high rank, it just "riding co-tails" of other strong pages.

From what you`ve described, you see some of these too ( " take over 1 or more of the page one results in my niche...") For these, if the 1st page gets removed, another will probably take its place...

These are pretty "strong" domains we are talking about with enormous link profiles, tons of social signals, a lot of visits etc.. You can probably "beat them" at some niche KW, but this will take some work.
This is somewhat like competing  against Wikipedia, which dominates a lot of niche KW and some major KW too...
As to your second question, permanent links will provide permanent value.

Had a lot of experience with this... What you need to know here is:

A. ) Remember that Google will always see/cache the page as seen by user without any cookie.

B. ) This is an acceptable method of doing thing (personalization is allowed) but I wouldn't create 2 totally different content pages here. Small changes are fine but anything major can be considered as "hidden content".

C.) As a rule, using different URLs is better for SEO (but maybe not for conversion) For example, I had a site that used cookie for language setting - simply spreading the whole content over unique URL improved organic traffic by+100% (and we are talking many thousands here). Not saying this is you case, because your description sound more like a landing-page optimization, but  wanted to mention this for any future thread readers.

GL

Thanks!

Very helpful. (+1)

Google officially said the site speed is a SEO factor which, when implemented, will affect 1% of SERP results. This gives some idea of the scope of it's relevancy. (not saying 1% is either high or low, this is all very subjective)

Having said that, beyond speed improvement, CDN provides other SEO related benefits.

I've actually just published a 2-part Blog post that investigates SEO & CDN related myths and talks about CDN and SEO benefits, I hope you`ll find it useful. it should answer all questions:

Part 1: SEO & CDN Myths
Part 2: SEO & CDN benefits

This is an issue I`ve dealt with over and over again for Joomla sites.

I had some good results with Canonical URL Module but didnt use it yet for 2.5 so Im reluctant to suggest it.

Im currently working with our Dev. Team to solve this for K2 - as far as I know, if you are a K2 user youll need a custom solution.

(yes Im aware of K2 canonical plugin and no, I dont recommend it)

GL

I`ve recently covered this in a blog post about SEO & CDN Myth busting. The short answer is "No", as Google is well aware of CDN structures.
I think the image traffic drop is un-related and it may have something to do with Bot Blocking features that prevent access from Google Image bot (which, as we recently discovered, can use non-US IPs and thus may also be considered as "fake" ).

Having said that, Google official statement says that speed factors only affect 1% of SERP results. If your image traffic is high or/and valuable I would re-think this.

60-70$ a month is way too much. Today you can get free Cloud CDN + Proxy acceleration from a number of providers.

Hi, I work at Incapsula and I've just recently wrote a 2 part Blog Post that look and CDN from a SEO point-of-view. The first post deals with  SEO & CDN Myths and the second one covered CDN related SEO Benefits. I hope you`ll find this helpful.

Admittedly, I do work for a CDN provider, but I also have 7+ year of prior SEO experience (both as Freelancer, Senior Adviser, CTO and SEO/SEM Department Manager) and so, in my post, I insisted on conduction balanced discussion, using only Google official statements to prove my point.

Hope this helps.