Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Stop Google crawling a site at set times

Intermediate & Advanced SEO

985

ske11 last edited by

Hi All

I know I can use robots.txt to block Google from pages on my site but is there a way to stop Google crawling my site at set times of the day? Or to request that they crawl at other times?

Thanks

Sean
1 Reply Last reply
Reply Quote 0
ske11 @ske11 last edited by

Thanks Keri, this gives me a place to start
1 Reply Last reply
Reply Quote 0
KeriMorgret @ske11 last edited by

Here's a link to Bing's help page on the tool. http://www.bing.com/webmaster/help/crawl-control-55a30302.
1 Reply Last reply
Reply Quote 1
ske11 @KeriMorgret last edited by

Thanks for that Keri, do you know what Bing calls that option , it might give me a starting point for a Google search

Sean
2 Replies Last reply
Reply Quote 0
KeriMorgret last edited by

I know Bing has a way to do that, but I haven't heard of Google releasing a similar tool.
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

If I deindex a page then will Google stop counting those links pointing to it?

Hey everyone, I am deindexing some posts of my website as I think they are not providing any value to the users. My question is that if I deindex a post and it has some good quality links pointing to it, will google stop those links counting for my website?
Intermediate & Advanced SEO | | Bunnypundir

0
Blocking Certain Site Parameters from Google's Index - Please Help

Hello, So we recently used Google Webmaster Tools in an attempt to block certain parameters on our site from showing up in Google's index. One of our site parameters is essentially for user location and accounts for over 500,000 URLs. This parameter does not change page content in any way, and there is no need for Google to index it. We edited the parameter in GWT to tell Google that it does not change site content and to not index it. However, after two weeks, all of these URLs are still definitely getting indexed. Why? Maybe there's something we're missing here. Perhaps there is another way to do this more effectively. Has anyone else ran into this problem? The path we used to implement this action:
Google Webmaster Tools > Crawl > URL Parameters Thank you in advance for your help!
Intermediate & Advanced SEO | | Jbake

0
Best Practice for setting up expert author contributing to Multiple Sites?

If a single author contributes to multiple sites, should each site have its own author page (tying to the same single gg+ account)? Ex. One author > one gg+ account > multiple author pages (one per site) Or, should all sites publishing his content link to a single author page/bio on a single, main site? Ex. One author > one gg+ account > a single author page on one site (all other sites link to this author page) In this event, where would the 'contributor to' link point for the additional sites he is contributing to, the homepage? Thanks!
Intermediate & Advanced SEO | | seagreen

0
Google Penalty or Not?

One of my sites I work with got this message: http://www.mysite: Unnatural inbound linksJune 27, 2013 Google has detected a pattern of artificial or unnatural links pointing to your site. Buying links or participating in link schemes in order to manipulate PageRank are violations of Google's Webmaster Guidelines. As a result, Google has applied a manual spam action to mysite.com/. There may be other actions on your site or parts of your site. But, when I got to manual actions it says: Manual Actions No manual webspam actions found. -- So which is it??? I have been doing link removal, but now I am confused if I need to do a reconsideration request or not.
Intermediate & Advanced SEO | | netviper

0
Buying a Google News website, against Google Terms?

We are looking at buying a business that has a number of websites Is it against buying a business that has a Google News website and continue to use the site? Once the business is sold, would google remove the site from its News?
Intermediate & Advanced SEO | | JohnPeters

0
How would I know if Google is showing me as two separate sites?

I work for a company that is (for example) www.neat-stuff.com, most people just type in www.neatstuff.com. I think that we are being counted as a site twice. Any suggestions as to how to know for sure? If you want to know why I am asking this here is the link http://www.seomoz.org/q/redirect-help
Intermediate & Advanced SEO | | EcommerceSite

0
We are ignored by Google - what should we do?

Hi, We believe that our website - https://en.greatfire.org - is being all but ignored by Google Search. The following two examples illustrate our case. 1. Searching for “China listening in on Skype - Microsoft assumes you approve”. This is the title of a blog post that we wrote which received some 50,000 visits. On Yahoo and Bing search, we rank first for this search. On Google, however, we rank 7th. Each of the six pages ranking higher than us are quoting and linking to our story. 2. Searching for “Online Censorship In China”. This is the title of our front page. Yahoo and Bing both rank us third for this search. On Google, however, we are not even among the first 300 results. Two of the pages among the first 10 results link to us. Our website has an average of around 1000 visits per day. We are quoted in and linked from virtually all Western mainstream media (see https://en.greatfire.org/press). Yet to this day we are receiving almost no traffic from Google Search. Our mission is to bring transparency to online censorship in China. If people could find us in Google, it would greatly help to spread awareness of the extent of Internet restrictions here. If you could indicate to us what the cause of our poor rankings could be, we would be very grateful. Thank you for your time and consideration.
Intermediate & Advanced SEO | | GreatFire.org

0
Google Places

If you rank on google places, I have noticed that you do not rank on the front page as well. I have a site that ranks on front page for it's keywords; however, because they are (1) on google places, they don't show up when someone is localized to that area. They show up on google places but not on front page. If you turn of localization, they are first in serps. How can I get around this? Two separate sites? One for Google+ (Places) and one for SERPS?
Intermediate & Advanced SEO | | JML1179

0