Google crawl rate dropped after we activated CloudFront
-
Hello!
Previously we've been using Amazon CloudFront for our static content (js, css etc). But to be able to reduce load on our origin servers and to be able to give our international users a good user experience we decided to deliver a couple of our sites through CloudFront.
We noticed very nice drops in page load time, but when checking Google webmaster tools we noticed that all CloudFront-activated sites got a huge drop in pages crawled per day (from avg ~3500 to ~150). Also one of the sites have issues with the Google sitemaps (just marked as "Pending" in GWT) and no new pages or updated pages seems to be updated in the Google SERP. The rest of the sites gets some updates on the Google SERP, but very few compared to before CloudFront activation.
Is there anybody here who have experience in full site delivery through CloudFront (or other CDNs) and effects on SEO/Google? Would be very glad for any insights or suggestions. The risk is that we need to remove CloudFront if this just continues.
-
My very best educated guess (after talking with dev friends here at Moz) is that CloudFront is supposed to be used for long-term storage. This means the content isn't expected to change frequently. This is great for images and other static things. However, this is also a signal to Google not to crawl your site as often as it won't be changing that often.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Ecommerce Alerts
I recently started getting email notifications from Google re: new products on our websites. I am subscribed to Google alerts. Can anyone shed some light on this?
Technical SEO | | AMHC0 -
Google Crawling Issues! How Can I Get Google to Crawl My Website Regularly?
Hi Everyone! My website is not being crawled regularly by Google - there are weeks when it's regular but for the past month or so it does not get crawled for seven to eight days. There are some specific pages, that I want to get ranked but they of late are not being crawled AT ALL unless I use the 'Fetch As Google' tool! That's not normal, right? I have checked and re-checked the on-page metrics for these pages (and the website as a whole, backlinking is a regular and ongoing process as well! Sitemap is in place too! Resubmitted it once too! This issue is detrimental to website traffic and rankings! Would really appreciate insights from you guys! Thanks a lot!
Technical SEO | | farhanm1 -
Website being crawled but not indexed any thoughts?
Hi Everyone,
Technical SEO | | Ant71
I created a new website a few weeks ago www.drivingseaford.co.uk , did a little link citation, links from Google+, submitted to webmaster tools etc but its still not getting indexed. Webmaster tools crawl stats page is showing pages being crawled, no errors. But 0 indexed. http://www.drivingseaford.co.uk/robots.txt is showing User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Im a bit stumped as never had this before!!! Any ideas from you lovely people?? Antony0 -
Google Dancing?
Hello, I was wondering why my website for some keywords goes from 2nd 3rd page in Google to 7th or even more sometimes? This happens since a while. Any suggestion? Thanks. Eugenio
Technical SEO | | socialengaged0 -
Google authorship what am i doing wrong
Hi, i am using google authorship on my site but when i use the testing tool it is not working. before the upgrade we had it working fine but now it does not seem to work. we have our google plus account pointing to the site and the writer we are trying to add is not coming up on the tool here is the code we are putting on the page Google+ and the page in question is here http://www.in2town.co.uk/emmerdale/emmerdale-laurel-is-determined-to-take-action when i check the tool i get the following Authorship Testing ResultAuthorship is not working for this webpage.andAuthorship rel=author MarkupCannot verify that rel=author markup has established authorship for this webpage.the tool i am using to check is http://www.google.com/webmasters/tools/richsnippetsany help to solve this problem would be great. i am using joomla
Technical SEO | | ClaireH-1848860 -
I have 3500 pages crawled by Google, - why is SEOMOZ only able to crawl 400 of these ?
I added my site almost two weeks ago to the PRO DashBoard, and so far only 404 pages has been crawled, - but I know for a fact that there is 3500 pages that should be crawled. Other search engines has no problem in crawling and indexing these pages, so what can be wrong here ?
Technical SEO | | haybob270 -
About google Disavow tool
My website is attacked by spammed link method, so should i use Goolge disavow tool to remove that links? And i have an question that when i use google Disavow to remove backlinks, but i still not remove it on the webpage that placed my links. Does Google index that backlink again? or never?
Technical SEO | | magician0 -
Having trouble removing homepage from google
For various reasons my client wants their homepage removed from google, no just the content of the page off but the page not to be indexed (yep strange request but we are mere service providers) today I requested in webmaster tool that default.asp was removed. Wht says done but the sites homepage is still listed. The page also has a no index tag on but 24 hours and 18k Google bot hits later it still remains. Anyone got any other suggestions to deindex just the homepage asap please
Technical SEO | | Grumpy_Carl0