Good answer Ben.

My main site is my own CMS, that I built 10 years ago, so after I added a lot of things to the .htaccess file and it became too large, I just moved the handling inside the control program, that only looks up filed URLs when they are broken. This processing is fast, but if there was any degradation, it only affects the broken URLs.

Speaking of broken URLs, I was getting a few 400 return codes and it seems the webserver handles those, so you have no chance to handle it in .htaccess. So the wat to handle that is with a 400 handler - that on cpanel sites just needs a 400.shtml file, that you can customize.

• you get a 400 response if you request a URL with a % symbol on the end, and some other site did that, thanks very much, and then google decided it would be a great thing to index.

Thank you David,

I didn't know that!

Thank you Edward.

I don't have quite that problem, but I think you are right too.

My CDN is set up to be Origin Pull.

That means there is no need to FTP - the system just fetches content as requested.

• you should check that out if you have to ftp everything.

But what you said that helped me is this - that I should have had one CNAME for images and anotehr CNAME for content and the content should be limited to a folder called content, so I can put the CSS files and the JS files in it and that way, the plain HTML pages at teh root level will never be affected.

I also realized, while checking the system, that I wasn't using a canonical tag in the intermediate pages, as I was in the story pages. So I just added code to add canonical tags for all the intermediate pages and the front page.

I do have a few other types of pages, so I will handle the code for them next.

I think adding the canonical tag might fix the problem, but I will also work on reconfiguring the CDN and change over when the action is not too busy, in case it takes a while to propagate.

Just when I thought I'd seen it all, google hits me with another nasty surprise!

I have a CDN to deliver images, js and css  to visitors around the world. I have no links to static HTML pages on the site, as far as I can tell, but someone else may have - perhaps a scraper site?

Google has decided the static pages they were able to access through the CDN have more value than my real pages, and they seem to be slowly replacing my pages in the index with the static pages.

Anyone got an idea on how to stop that?

Obviously, I have no access to the static area, because it is in the CDN, so there is no way I know of that I can have a robots file there.

It could be that I have to trash the CDN and change it to only allow the image directory, and maybe set up a separate CDN subdomain for content that only contains the JS and CSS?

Have you seen this problem and beat it?

(Of course the next thing is Roger might look at google results and start crawling them too, LOL)

P.S. The reason I am not asking this question in the google forums is that others have asked this question many times and nobody at google has bothered to answer, over the past 5 months, and nobody who did try, gave an answer that was remotely useful. So I'm not really hopeful of anyone here having a solution either, but I expect this is my best bet because you guys are always willing to try.

Hello Zeal Digital,

I use a CDN (Content Delivery Network) for images, CSS and javascript.

Doing that adds only about $10 to cost per month for a site that had around 800,000 pageviews per month.

You have complete control over the images. If there is a problem, you can force the CDN to flush a file and reload it from the source. You add code to your .htaccess file that tells the CDN how long to store images before fluching them and getting a new copy. It is all automated, there is generally no work for you to do. I host with softlayer.com and this is part of their service.

The change from self-sourced images, css and scripts had a massive improvement on the server.

• it is a 16-processor linux box with twin 15,000rpm SCSI drives and 12Gb RAM - it is quite fast!

Page delivery times improved by 1-2 seconds.

The server now is so lightly loaded that it could be downgraded to save more money.

It has zero effect on SEO. The CDN is accessed using a CNAME.

-   static.domain.com  - so don't worry about it looking like components are from other places.

The CDN has servers all over the world, so no matter where the visitors are, it is only a few hops for them to get most of the content, making it much faster for someone in Australia who would normally pull images from a server in the USA.

Your only problem with Amazon S3 is that they have crashed it a few times, but other than that, it is a good thing to do.

I wouldn't advise them to self-host, unless you want to increase their costs, server loading and page delivery times.