Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Stop Google crawling a site at set times

Intermediate & Advanced SEO

985

ske11 last edited by

Hi All

I know I can use robots.txt to block Google from pages on my site but is there a way to stop Google crawling my site at set times of the day? Or to request that they crawl at other times?

Thanks

Sean
1 Reply Last reply
Reply Quote 0
ske11 @ske11 last edited by

Thanks Keri, this gives me a place to start
1 Reply Last reply
Reply Quote 0
KeriMorgret @ske11 last edited by

Here's a link to Bing's help page on the tool. http://www.bing.com/webmaster/help/crawl-control-55a30302.
1 Reply Last reply
Reply Quote 1
ske11 @KeriMorgret last edited by

Thanks for that Keri, do you know what Bing calls that option , it might give me a starting point for a Google search

Sean
2 Replies Last reply
Reply Quote 0
KeriMorgret last edited by

I know Bing has a way to do that, but I haven't heard of Google releasing a similar tool.
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

XML sitemap generator only crawling 20% of my site

Hi guys, I am trying to submit the most recent XML sitemap but the sitemap generator tools are only crawling about 20% of my site. The site carries around 150 pages and only 37 show up on tools like xml-sitemaps.com. My goal is to get all the important URLs we care about into the XML sitemap. How should I go about this? Thanks
Intermediate & Advanced SEO | | TyEl

0
Why this site is not hit by google penguin only experts answer please

Hello I know a site which has the following things and it is not hit by google penguin at all and is ranked on first page first link in google.com.pk it has 1.3K total backlinks its has 34 referring domains according to ahrefs the exact match keyword which is ranked in google.com.pk as first position is used in 1.2 K backlinks its DA is 18 and PA is 31 its citation flow is 19 trust flow is 13 targeted exact match keyword is present in Title and description also. Now tell me what is happening here.
Intermediate & Advanced SEO | | tanveerayakhan

0
Dfferent url of some other site is shown by Google in cace copy of our site's page

Hi, When i check cached copy of url of my site http://goo.gl/BZw2Zz , the url in cache copy shown by Google is of some other third party site. Why is Google showing third party url in our site's cached url. Did any of you guys faced any such issue. Regards,
Intermediate & Advanced SEO | | vivekrathore

0
Moving career site to new URL from main site. Will it hurt SEO for main page?

For one of our clients we are building a career site and putting it under a different URL and hosting service (mainly due to security concerns of hosting it under the same host and domain). almost 100% of the incoming traffic to their current career section (which it is in a sub-folder) receives traffic for branded keywords (brand + job/career/employment), that is, there are no job position specific keywords. The client is now worried that after moving the site, the inbound traffic to the main site will be severely affected as well as the SERP results. My questions are, will the non-career related SERPs be affected? I don't see how will they be but I could be wrong If no, how could we reassure her that the SEO to the main site wont be affected? are there any case studies of a similar case (splitting part of the website under a new URL and hosting service?) Thank you for your help. PS: this is my first post so please forgive me if this has been asked before. I could not find a good response.
Intermediate & Advanced SEO | | rflores

0
Why are these sites outranking me?

I am trying to rank for the phrase "a link between worlds walkthrough" I am on page 1 but there are several results that just outranks me and I cannot see any reason that they would be doing so. My site is hiddentriforce.com/a-link-between-worlds/walkthrough/ For that page I have 5 linking domains, varied anchor text that spans from things like "here" to a variety of related phrases. All of the links come from really good sites My page has 1400 likes, 90 shares, and about 20 each in tweets and +'s DA of 44 PA of 37 The 4 and 5 ranked sites both have WAY less social interactions, lower PA and DA, less links, etc Yet they outrank me why?
Intermediate & Advanced SEO | | Atomicx

0
What is wrong with my site?

I could use suggestions/input. My site is consistently being beaten by doorway pages with zero to no content. In fact, on the keyword "lawrenceville plumber" I am being beaten by a KID with a blank wordpress installation. I really need help determining what my issues are and what I can do to help. According to all of the graders, ranks, etc my site is great. I have written tons of unique content, have added a blog unique articles and self-helps. Please help! My site is www.akinsplumbing.net. Thanks!
Intermediate & Advanced SEO | | chuckakins

0
Google giving me only partial site links?

Hi Guys, My site is #1 ranked for the term "waiting till marriage," but Google only gives me partial site links. See "Forums - Articles - Questions - Videos" links in attached screenshot. How do I get the full, page-dominating, mini-description-having site links? Any suggestions? Note: I've got a ton of content and decent traffic, but I haven't put much time into developing back links yet. I'm a php developer, but I'm new to professional-level SEO. Any help would be hugely appreciated. Also, sorry about the inflammatory nature of the site. It's not a preachy site; it's just a support group. Hope it doesn't offend. partial-sitelinks.png
Intermediate & Advanced SEO | | MikeAM27

0
Getting Google to Correct a Misspelled Site Link...Help!

My company website recently got its site links in google search... WooHoo! However, when you type TECHeGO into Google Search one of the links is spelled incorrectly. Instead of 'CONversion Optimization' its 'COversion Optimization'. At first I thought there was a misspelling on that page somewhere but there is not and have come to the conclusion that Google has made a mistake. I know that I can block the page in webmaster tools (No Thanks) but how in the crap can I get them to correct the spelling when no one really knows how to get them to appear in the first place? Riddle Me That Folks! sitelink.jpg
Intermediate & Advanced SEO | | TECHeGO

0