Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Set Robots.txt file to crawl my website at specific times

Intermediate & Advanced SEO

401

CharleneKennedy12 last edited by

Our website provider has stated that they can only 'lift' their block on our website in order for it to be crawled as specific times.

Is there any way to amend a robots.txt to ensure that it crawls our website at a specific time of day/night in order to coincide with the block being lifted?

Many Thanks,

Charlene
1 Reply Last reply
Reply Quote 0
Tenlo last edited by

Charlene,

Your question does not make any sense, what block are they talking about? If your website provider is not allowing Google to crawl your site you should go look at getting a new provider straight away.
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Setting country specific top level domain as alias - will site benefit from TLDs authority?

I have a host of sites that follow a top level domain strategy. For each local site they will be on the top level domain but with their country-languages prefix as the subdirectory. Such as below: example.com
example.com/uk-en
example.com/sg-en
example.com/de-de Each local site being on the TLD will benefit them in terms of SEO and it makes it easier to have one strategy. My question however, if the Netherlands comes on board, they would generally have example.com/nl-en. However they want their primary domain as examplenetherlands.nl and the TLD (example.com/nl-en) set as an alias/secondary domain that redirects to the primary. Will they benefit from any SEO if the TLD is not the primary address?
Intermediate & Advanced SEO | | gracejo

0
International website. Di I need a new website

i am looking to expand from the UK and open a location in the US. i curretly have a .co.uk domain. what would you recommend I do with th website, create a new one wth a .com domain?
Intermediate & Advanced SEO | | Caffeine_Marketing

0
SEO - Use pages on main site or set up outside keyword rich domains and websites

I have a client who is wanting to target searches for competitors products. His idea was to purchase domains related to the searches he's targeting (for example, people looking for another company's app) and to build out one page websites addressing the search query and why a customer would choose his app solution over a competitor. I know he'd have to build a handful of links to each site for any chance of success but I wanted to ask the following.. Would doing this be better than just building pages addressing the searches on his main website domain? Is there an SEO risk to doing this? Potential for a penalty? Anything we need to do to structure these in a way that won't violate Google's SEO guidelines? Any other thoughts on pros and cons of each strategy? Thank you! Ricky
Intermediate & Advanced SEO | | RickyShockley

0
Twitter Robots.TXT

Hello Moz World, So, I trying to wrap my head around all of the different robots.txt. I decided to dive into a site like Twitter, and look at their robot text. And now, I'm super confused. What are they telling the search engines with /hasttag/*src=. Why don't they just use: Useragent: * Disallow: But, they address each search engine. Is there any benefit to this? Thanks for all of the awesome responses!!! B/R Will H.
Intermediate & Advanced SEO | | MarketingChimp10

0
SSL and robots.txt question - confused by Google guidelines

I noticed "Don’t block your HTTPS site from crawling using robots.txt" here: http://googlewebmastercentral.blogspot.co.uk/2014/08/https-as-ranking-signal.html Does this mean you can't use robots.txt anywhere on the site - even parts of a site you want to noindex, for example?
Intermediate & Advanced SEO | | McTaggart

0
Can I use the old website content on the new website, after deleting it from the server?

My website nowwhatstudio.com hit by google pure spam and google applied manual spam action to the website. I create new website (nowwhatmoments.com) with the same content from the old spam action website (nowwhatstudio.com). As google removed my old website content from search indexed. Can I use the same content for a new website? If I delete my old website from the server, after that Can I use the old website content for a new website? Or Can make edits the old website content and make it 80% original for a new website?
Intermediate & Advanced SEO | | bondhoward

0
Specific KW question...

Hi, I have this site: http://www.aerlawgroup.com. It's ranking very well overall for all targeted KWs. However, I have seen a drop for one main KW: "Los Angeles criminal defense attorney." It currently ranks #8 (it used to be as high as #2). What's interesting is that for similar (yet slightly less competitive KWs, he ranks much better - "Los Angeles Criminal Defense Lawyer." I'm not trying to be greedy with rankings, but I would love feedback and/or tips regarding any issues that could be contributing to this drop. Thanks.
Intermediate & Advanced SEO | | mrodriguez1440

0
Bing seriously hitting our website

Hi I have a strange query, Bing in the last four weeks have been seriously crawling our site, and while at the moment this isn't affecting our servers, coming into the busy time of the year I am just wondering if anyone else is seeing this. They are basically crawling the entire site (including nofollow pages) even though according to Bing Webmaster tools, they know our sitemap. Is anybody else seeing any unusual activity with the Bing bot. Thanks Andy
Intermediate & Advanced SEO | | Andy-Halliday

0