Suggested crawl rate in google webmaster tools?
-
hey moz peeps, got a general question:
what is the suggested custom crawl rate in google webmaster tools? or is it better to "Let Google determine my crawl rate (recommended)"
If you guys have any good suggestions on this and site why that would be very helpful, thanks guys!
-
Hi David,
According to Google's webmasters help page, it states that the crawl rate option is just about how fast it crawls your site and not how often it does. Check out the link for more information about it.
Also, I wouldn't stress to much on the settings because the search engines are going to do what they do, just make sure to create good content and get it out there, the search engines will see this and depending on how well it is they we come more often on there own.
Hope this helps!
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=48620
-
thanks for that alan, yea id like to see what other have to say about this as well, im all open ears!
-
I believe it is much better to allow the search engines to handle it themselves. Check your logs to see what they are doing, so you know if they are hammering you server. If they are, you can slow them down using the robots.txt file On a large site, google will often such out a few hundred thousand pages per day. Yahoo will do more, but their crawler isn't always friendly or smart. Sometimes different Slurp crawlers will request the same pages. Bing emailed me six months ago and asked that we remove the 2second delay I added. I did that because they were hammering the server, but only sent a few thousand readers per day. Google doesn't usually hammer the server, but when 5 or 6 are hitting you at the same time, it can affect you site response time. - and then some of them want to penalize you for having a poor experience for visitors.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website crawl error
Hi all, When I try to crawl a website, I got next error message: "java.lang.IllegalArgumentException: Illegal cookie name" For the moment, I found next explanation: The errors indicate that one of the web servers within the same cookie domain as the server is setting a cookie for your domain with the name "path", as well as another cookie with the name "domain" Does anyone has experience with this problem, knows what it means and knows how to solve it? Thanks in advance! Jens
Technical SEO | | WeAreDigital_BE0 -
Clarification on indexation of XML sitemaps within Webmaster Tools
Hi Mozzers, I have a large service based website, which seems to be losing pages within Google's index. Whilst working on the site, I noticed that there are a number of xml sitemaps for each of the services. So I submitted them to webmaster tools last Friday (14th) and when I left they were "pending". On returning to the office today, they all appear to have been successfully processed on either the 15th or 17th and I can see the following data: 13/08 - Submitted=0 Indexed=0
Technical SEO | | Silkstream
14/08 - Submitted=606,733 Indexed=122,243
15/08 - Submitted=606,733 Indexed=494,651
16/08 - Submitted=606,733 Indexed=517,527
17/08 - Submitted=606,733 Indexed=517,498 Question 1: The indexed pages on 14th of 122,243 - Is this how many pages were previously indexed? Before Google processed the sitemaps? As they were not marked processed until 15th and 17th? Question 2: The indexed pages are already slipping, I'm working on fixing the site by reducing pages and improving internal structure and content, which I'm hoping will fix the crawling issue. But how often will Google crawl these XML sitemaps? Thanks in advance for any help.0 -
Will blocking the Wayback Machine (archive.org) have any impact on Google crawl and indexing/SEO?
Will blocking the Wayback Machine (archive.org) by adding the code they give have any impact on Google crawl and indexing/SEO? Anyone know? Thanks! ~Brett
Technical SEO | | BBuck0 -
Has Google Stopped Listing URLs with Crawl Errors in Webmaster Tools?
I went to Google Webmaster Tools this morning and found that one of my clients had 11 crawl errors. However, Webmaster Tools is not showing which URLs are having experiencing the errors, which it used to do. (I checked several other clients that I manage and they list crawl errors without showing the specific URLs. Does anyone know how I can find out which URLs are experiencing problems? (I checked with Bing Webmaster Tools and the number of errors are different).
Technical SEO | | TopFloor0 -
Http VS https and google crawl and indexing ?
Is it true that https pages are not crawled and indexed by Google and other search engines as well as http pages?
Technical SEO | | sherohass0 -
Will Google index a site with white text? Will it give it bad ratings?
Will google not rank a site bc pretty much all the copy is white (and the background is all white)? Here's the site in question: https://www.dropbox.com/s/6w24f6h5p0zaxhg/Garrison_PLAY.vs2-static.pdf https://www.dropbox.com/sh/fwudppvwy2khpau/t43NozpG3E/Garrison_PLAY.vs3.jpg thanks--if you need me to clarify more let me know TM Humphries LocalSearched.com
Technical SEO | | CloudGuys0 -
Google Fresh Update
Here's a question, if you put the date in your url (often a wordpress format) would that be seen by Google now as 'fresh' content? - part of their new update.
Technical SEO | | pauledwards0 -
Google Penalize?
Hello, I read an statement somewhere which stated: "2 identical URLs linked to 2 different popular key phrases next to each other (on the same website/domain) will lead to a Google penalize. Google knows, that both terms are popular. This means, Google will ignore the links to your site (you'll not have any benefit) and the site you have your links on loses authority." What are your thoughts on this statement? Thank you.
Technical SEO | | micfo0