Crawl rate
-
Hello,
In google WMT my site has the following message.
<form class="form" action="/webmasters/tools/settings-ac?hl=en&siteUrl=http://www.prom-hairstyles.org/&siteUrl=http://www.prom-hairstyles.org/&hl=en" method="POST">Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate.Why would this be?A bit of backgound - this site was hammered by Penguin or maybe panda but seems to be dragging itself back up (maybe) but has dropped from several thousand visitors/day to 100 or so.Cheers,Ian</form>
-
OK thanks.
Ian
-
There is nothing negative about the message. It is common and expected for sites which use CDNs.
-
Thanks for replying, I have been using a CDN (up until a week ago) and no there haven't been any issues that I can put down to this, I just hadn't seen a message like that before and wondered if it was significant.
-
What is your hosting setup? Are you using some form of cloud hosting such as Amazon or have a CDN set up?
Generally speaking, that message from Google is a good thing. It means they have optimized your crawl rate based on your setup. If you feel there is a need to adjust the rate, it can be done. Are you experiencing any issue?
The Panda/Penguin issue would have no relation to your crawl rate settings.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Brushing up on my SEO skills - how do I check my website to see if Javascript is blocking search engines from crawling the links within a javascript-enabled drop down menu?
I set my user agent in my Chrome browser to Googlebot and I disable javascript within my Chrome settings, but then what?
Technical SEO | | MagnitudeSEO0 -
Crawl errors which ones should i sort out
Hi, just had my website updated to joomla 3.0 and i have around 4000 urls not found. now i have been told i need to redirect these but i would just like to check on here to make sure i am doing the right thing and the advice i have been given is not correct. I have been told these errors are the reason for the drop in rankings. I need to know if i should redirect all of these 4,000 urls or only the ones that are being linked to from outside of the site. I think about 3,000 of these have no links from outside of the site, but if i do not redirect them all then i am going to keep getting the error messages. around 2,000 of these url not found are from the last time we updated the site which was a couple of years ago and i thought they would have died off now. any advice on what i should do would be great
Technical SEO | | ClaireH-1848860 -
Changes to website haven't been crawled in over a month
We redesigned our website at http://www.aptinting.com a few months ago. We were fully expecting the crawl frequency to be very low because we had redesigned the website from a format that had been very static, and that probably has something to do with the problem we're currently having. We made some important changes to our homepage about a month ago, and the cached version of that page is still from April 2nd. Yet, whenever we create new pages, they get indexed within days. We've made a point to create lots of new blog articles and case studies to send a message to Google that the website should be crawled at a greater rate. We've also created new links to the homepage through press releases, guest blog articles, and by posting to social media, hoping that all of these things would send a message to Google saying that the homepage should be "reevaluated". However, we seem to be stuck with the April 2nd version of the homepage, which is severely lacking. Any suggestions would be greatly appreciated. Thanks!
Technical SEO | | Lemmons0 -
Crawl Diagnostic: Notices about 301 redirects
There are detected five 301 redirects on my site and I want to understand why this is happening? And is this important to fix? http://domain.cl/subfolder ---- redirects to ----> http://domain.cl/subfolder/ What does this tell me "/" I am very curious 🙂 Thanks for every answer
Technical SEO | | inlinear
Holger0 -
Can too many pages hurt crawling and ranking?
Hi, I work for local yellow pages in Belgium, over the last months we introduced a succesfull technique to boost SEO traffic: we have created over 150k of new pages, all targeting specific keywords and all containing unique content, a site architecture to enable google to find these pages through crawling, xml sitemaps, .... All signs (traffic, indexation of xml sitemaps, rankings, ...) are positive. So far so good. We are able to quickly build more unique pages, and I wonder how google will react to this type of "large scale operation": can it hurt crawling and ranking if google notices big volumes of content (unique content)? Please advice
Technical SEO | | TruvoDirectories0 -
What does the Google Crawler see when crawling this page?
If you look at this page http://www.rockymountainatvmc.com/t/49/61/185/730/Batteries. You will see we have a vehicle filter on it. Right now you only see a picture of a battery and some bad text that needs to be updated ( We just hired a copywriter!). Our question is when google crawls this site will thy just see this or will they see all the products that appear after you pick a "machine type" "make" "model" and "year" Any help would be great. Right now we think it just sees this main page how we have set things up; however, we know that the crawler is also crawling some ajax. We just want to be sure of things.
Technical SEO | | DoRM0 -
SEOMoz Crawling Errors
I recently implemented a blog using WordPress on our website. I didn't use WordPress as the CMS for the rest of our site just the blog portion. So as an example I installed Wordpress in http://www.mysite/blog/" not in the root. My error report in SEOMoz went from 0 to 22e. The Moz bot or crawler that SEOMoz uses is reporting a ton of 4xx errors to strang links that shouldn't exist anywhere on the site. Example: Good link - http://www.mysite/products.html Bad link reported by SEOMoz - http://www.mysite/blog/my-first-post/products.html I've also noticed that my page speed as become much slower as reported by Google. Does anybody know what could be happening here? I know that typically it's better to install WordPress in the root and use it to control the entire site but I was under the gun to get a blog out. Thanks
Technical SEO | | TRICORSystems0 -
Reading Crawl Diagnostics and Taking Action on results
My site crawl diagnostics are showing a high number of duplicate page titles and content. When i look at the flagged pages, many errors are simply listed from multiple pages of product category search results. This looks pretty normal to me and I am at a loss for understanding how to fix this situation. Can I talk with someone? thanks, Gary
Technical SEO | | GaryQ0