How to Set Custom Crawl Rate in Google Webmaster Tools?
-
This is really silly question to set custom crawl rate in Google webmaster tools. Any one can find out that section under setting tab. But, I have confusion to decide number for request per second and second between requests text field.
I want to set custom crawl rate for my eCommerce website.
I checked my Google webmaster tools and find out as attachment. So, Can I use this facility to improve my crawling?
-
Hi Daniel,
I was thinking about it due to very less number of indexing. Thanks for your suggestion.
-
It's better to just leave it on default - custom crawl rates are only needed if you're experiencing traffic problems from the server(as stated in your screenshot).
Yyou will also use more bandwidth on your own server by the google bot coming more often so this should also be something you think about. Also worth mentioning is that after 90 days it will go back to default settings.
http://googlewebmastercentral.blogspot.com/2006/10/learn-more-about-googlebots-crawl-of.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl diagnostic issue?
I'am sorry if my English isn't very good, but this is my problem at the moment: On two of my campagnes I get a weird error on Moz Analytics: 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag Moz Analytics points to an url that starts with: http:/**/None/**www.????.com. We don't understand how Moz indexed this non-existing page that starts with None? And how can we solve this error? I hope that someone can help me.
Intermediate & Advanced SEO | | nettt0 -
Site not indexed in Google UK
This site was moved to a new host by the client a month back and is still not indexed in Google UK if you search for the site directly. www.loftconversionswestsussex.com Webmaster tools shows that 55 pages have been crawled and no errors have been detected. The client also tried the "Fetch as Google Bot" tactic in GWT as well as running a PPC campaign and the site is still not appearing in Google. Any thoughts please? Cheers, SEO5..
Intermediate & Advanced SEO | | SEO5Team0 -
Implementation of AJAX Crawling Specifications
My URL is: http://www.redfin.com/TX/Austin/8413-Navidad-Dr-78735/home/31224372 We're using Google's AJAX crawling system, per the documentation here. https://developers.google.com/webmasters/ajax-crawling/The example page above requires JavaScript to display content; it includes in the source. We have a lot of pages like this on our site.We expect Google to query us at this URL:http://www.redfin.com/TX/Austin/8413-Navidad-Dr-78735/home/31224372?escaped_fragment=This page renders correctly with JavaScript disabled.Are we doing this correctly? There are some small differences between the escaped_fragment HTML snapshot and the JavaScript-generated content. Will this cause any problems for us?We ask because there was a period of about two months (from October 4th to Dec 29th) during which Google's crawler radically decreased the hits to our escaped_fragment URLs; it's maybe recovering now, but maybe it isn't, and I wanted to be absolutely sure we're doing this correctly.
Intermediate & Advanced SEO | | RyanOD0 -
How to remove an entire site from Google?
Hi people, I have a site with around 2.000 urls indexed in google, and 10 subdomains indexed too, which I want to remove entirely, to set up a new web. Which is the best way to do it? Regards!
Intermediate & Advanced SEO | | SeoExpertos0 -
Setting Up Google Analytics for domains with 301
I have a client with a google analytics account that is a mess. domaina.com domainb.com 302's to domaina.com domainc.com 302's to domaina.com domaind.com 302's to domaina.com I thought the client was doing 301s on all these domains to the primary domain. I have logged into there analytics account and found data is being tracked on the other domains i.e domainb,c,d.com etc. How is it possible that google analytics is tracking data on these domains when no analytics code has been created and the urls are redirecting to domaina.com? Also there are not sites on these domains so for webmaster tools should I enable domain verification through a cname on the dns? Also I can I best setup a way to track traffic coming from say domainb.com? Whats the best step by step guide to use to set this up.
Intermediate & Advanced SEO | | JohnW-UK0 -
Google Web Master Tools: Duplicate Title Tags?
According to Google Web Master Tools it says my site has 910 Duplicate Title Tags. How verbose should title tags be? What's the maximum character length? For example, let's say I have an image of an iPhone 4S being held in someone's hand. How verbose of a title and how many characters am I allowed to have? Is the goal with title tags to be very specific in describing the image? In the above example, do I need to say something like: "iPhone 4S being held by white caucasian male" or will "iPhone 4S" suffice?
Intermediate & Advanced SEO | | asc760 -
HELP - got the following message - Google Webmaster Tools notice of detected unnatural links
Hi All, While trying to grow we used several freelancers and small companies for guest blogging, article submissions etc. We lost about 90% of traffic from our peek at December. We don't know if it is related but we got the following message last week:
Intermediate & Advanced SEO | | BeytzNet
"Google Webmaster Tools notice of detected unnatural links to www.domain.com" Is it related (getting this message after two months of losing traffic)? What to do???? (P.S
We fired most of the companies we used months ago since we noticed they used bad methods. We didn't believe it can hurt us - just thought it would be useless...) Please Help...0 -
Get Higher in Google Shopping
Hello, A few days ago i imported my product list into Google Shopping and everything got accepted, but when i look in Google Shopping for my product. It's on page 3, how can i get my product higher in Google shopping? I assume this thing is different from just normal SEO? Regards, Yannick
Intermediate & Advanced SEO | | iwebdevnl0