Set Robots.txt file to crawl my website at specific times
-
Our website provider has stated that they can only 'lift' their block on our website in order for it to be crawled as specific times.
Is there any way to amend a robots.txt to ensure that it crawls our website at a specific time of day/night in order to coincide with the block being lifted?
Many Thanks,
Charlene
-
Charlene,
Your question does not make any sense, what block are they talking about? If your website provider is not allowing Google to crawl your site you should go look at getting a new provider straight away.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Meta robots
Hi, I am checking a website for SEO and I've noticed that a lot of pages from the blog have the following meta robots: meta name="robots" content="follow" Normally these pages should be indexed, since search engines will index and follow by default. In this case however, a lot of pages from this blog are not indexed. Is this because the meta robots is specified, but only contains follow? So will search engines only index and follow by default if there is no meta robots specified at all? And secondly, if I would change the meta robots, should I just add index or remove the meta robots completely from the code? Thanks for checking!
Intermediate & Advanced SEO | | Mat_C0 -
Allowing Guest Posts on Website
I am planning to allow Guest posts on my website blog where authors or writers can post articles to my blog. All content will be manually approved by me. But i would like to know if guest posts will cause any harm to my site or is it good way to generate content. Another thing as guest posts will be having do-follow links & author section too will be up. So firstly i would like to know whether its a good step or not? Would even like to know what checks should i do before approving a guest posts?
Intermediate & Advanced SEO | | welcomecure0 -
Redirects in .htaccess file
I am using yoast seo plugin for wordpress. I have done several link changes and placed approximately 80-100 301 redirect into .htaccsses file. Is this good practice? Do they have to be there forever? Is it not slowing down site? Or should this be done in different way? Thank you very much Iris
Intermediate & Advanced SEO | | Rebeca10 -
Large number of pages crawled.
My campaign for printlabelandmail.com says that seomoz has crawled 619 pages. My site, however, only has a little over 250 pages. Where are these extra pages? I did recently relaunched my website with wordpress. I was using Dreamweaver before. I thought I deleted all the old pages. Could these extra pages be old pages from the site prior to my relaunch? I hope my question makes sense. Any insights would be helpful. Thanks! Andrea
Intermediate & Advanced SEO | | JimDirectMailCoach0 -
About robots.txt for resolve Duplicate content
I have a trouble with Duplicate content and title, i try to many way to resolve them but because of the web code so i am still in problem. I decide to use robots.txt to block contents that are duplicate. The first Question: How do i use command in robots.txt to block all of URL like this: http://vietnamfoodtour.com/foodcourses/Cooking-School/
Intermediate & Advanced SEO | | magician
http://vietnamfoodtour.com/foodcourses/Cooking-Class/ ....... User-agent: * Disallow: /foodcourses ( Is that right? ) And the parameter URL: h
ttp://vietnamfoodtour.com/?mod=vietnamfood&page=2
http://vietnamfoodtour.com/?mod=vietnamfood&page=3
http://vietnamfoodtour.com/?mod=vietnamfood&page=4 User-agent: * Disallow: /?mod=vietnamfood ( Is that right? i have folder contain module, could i use: disallow:/module/*) The 2nd question is: Which is the priority " robots.txt" or " meta robot"? If i use robots.txt to block URL, but in that URL my meta robot is "index, follow"0 -
Backlinks from one website to my 3 websites (hosted in 1 c-block) ?
We are making some linkbuilding. And have very nice backlinks offer. So we are planning to put our 3 websites in it. Our 3 websites are on separate IP, but same C-block. Can it be a red flag for google? Can i put my 3 backlinks in one blog post?
Intermediate & Advanced SEO | | bele0 -
Websites with same content
Hi, Both my .co.uk and .ie websites have the exact same content which consists of hundreds of pages, is this going to cause an issue? I have a hreflang on both websites plus google webmaster tools is picking up that both websites are targeting different counties. Thanks
Intermediate & Advanced SEO | | Paul780 -
Website change of address
Hi Everyone, I apologize if the answer to this questions is obvious, but I wanted some input on how changing our web address of our site will affect our SERP. We are looking to change our website address from a.com to b.com due to rebranding of our company (primarly to expand our product line as our current url and company name are restricting). I understand that this can be done using 301 direct and via webmaster tools with google. My question is how does this work exactly? Will our old website address show in SERP rankings, and when a user clicks on the listing are they redirected to our new address? With regards to building new links from press releases etc, do we have links point to our new web address or the old one in order to increase SERP? Does google see our old address and new address as the same website and therefor it does not matter where inbound links point to and both will increase our ranking positions? It took 6 years of in house seo to get our website to rank on the first page of all the major search engines for our keywords, so we am being very cautious before we do anything. Thanks everyone for your input, it is greatly appreciated 🙂
Intermediate & Advanced SEO | | AgentMonkey0