Robots.txt Download vs Cache
-
We made an update to the Robots.txt file this morning after the initial download of the robots.txt file. I then submitted the page through Fetch as Google bot to get the changes in asap.
The cache time stamp on the page now shows Sep 27, 2013 15:35:28 GMT. I believe that would put the cache time stamp at about 6 hours ago. However the Blocked URLs tab in Google WMT shows the robots.txt last downloaded at 14 hours ago - and therefore it's showing the old file.
This leads me to believe for the Robots.txt the cache date and the download time are independent. Is there anyway to get Google to recognize the new file other than waiting this out??
-
No to my knowledge. You will have to wait. Anyway, Google could have already download the new robots but while the reports are showing the older file. Those reports always take a while until refreshing completely.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
DNS vs IIS redirection
I'm working on a project where a site has gone through a rebrand and is therefore also moving to a new domain name. Some pages have been merged on the new site so it's not a lift and shift job and so I'm writing up a redirect plan. Their IT dept have asked if we want redirects done by DNS redirect or IIS redirect. Which one will allow us to have redirects on a page level and not a domain level? I think IIS may be the right route but would love your thoughts on this please.
Technical SEO | | Marketing_Today1 -
One server, two domains - robots.txt allow for one domain but not other?
Hello, I would like to create a single server with two domains pointing to it. Ex: domain1.com -> myserver.com/ domain2.com -> myserver.com/subfolder. The goal is to create two separate sites on one server. I would like the second domain ( /subfolder) to be fully indexed / SEO friendly and have the robots txt file allow search bots to crawl. However, the first domain (server root) I would like to keep non-indexed, and the robots.txt file disallowing any bots / indexing. Does anyone have any suggestions for the best way to tackle this one? Thanks!
Technical SEO | | Dave1000 -
Http vs https: which is better for seo / usability
Hi All, Firstly thank you for taking the time to look. My dilemma is as follows; I have a site on wordpress that I have added an ssl certificate to and the entire domain is secure. The site has a mix of content including a blog area and product pages. My question is what does Google prefer, http or https or does it not matter As i see it my option is to keep the entire site as https and enforce this sitewide so all non secure content redirects to the https version or i could enforce https just in the cart and or product pages, all other content, homepage, blog, about us, contact us etc would be http. From an seo perspective ie google search engine, is their a best way to proceed? Finally, as i currently have http and https both displaying ie duplicate, what would be the way to fix this, i have yoast plugin so can set the canonical there and can also edit my robot.txt. I have come across this resource (http://www.creare.co.uk/http-vs-https-duplicate-content) and am wondering if this guideline is still correct or is there another more current way, if so I would be grateful if you could point me in the right direction. thanks in advance.
Technical SEO | | Renford_Nelson0 -
Pages not being cached have a negative effect?
Hi all! I look after a website where it's been discovered a section of the website has the noarchive robots meta tag active on it causing it to not get cached but has been indexed. Out of curiosity has anyone seen any negative effects from Google for having pages that aren't cached? It's not the strongest section on the website so makes it tricky to judge myself but interested if anyone had any thoughts on the matter. Cheers,
Technical SEO | | thisisOllie0 -
Googlebot does not obey robots.txt disallow
Hi Mozzers! We are trying to get Googlebot to steer away from our internal search results pages by adding a parameter "nocrawl=1" to facet/filter links and then robots.txt disallow all URLs containing that parameter. We implemented this late august and since that, the GWMT message "Googlebot found an extremely high number of URLs on your site", stopped coming. But today we received yet another. The weird thing is that Google gives many of our nowadays robots.txt disallowed URLs as examples of URLs that may cause us problems. What could be the reason? Best regards, Martin
Technical SEO | | TalkInThePark0 -
Blog archives vs individual articles
In a client's blog, you can find each individual article pages as well as aggregate of articles per month or sometimes per day (including each entire article). The problem is that the article appears twice, once in a dedicated page (article page) and once with other articles (in the archive). Is there a specific SEO approach to this type of situation? Is there duplicate content? What page name should I give each archive (if at all), as there are quite a few? Thank you
Technical SEO | | DavidSpivac0 -
Every time google caches our site it shows no website.
Our site <cite>www.skaino.co.uk/</cite> seems to be having real issues with being picked up with Google. The site has been around for a long time but no longer even ranks on google if you search for the word 'Skaino'. This is odd as its hardly a competitive keyword. If I do a site:www.skaino.co.uk then it shows all the pages proving the site has been indexed. But if I do cache:www.skaino.co.uk it shows a blank cache. I'm starting to worry that Google isn't able to crawl our site properly. If it helps to clarify we have a flash site with a HTML site running underneath for those who cant view flash. Im wandering if I've missed something glaringly obvious. Is it normal to have a blank google cache? Thanks AJ
Technical SEO | | handygammon0 -
WordPress Pretty Permalinks vs Site Speed
A couple of issues at play here as I wrestle with the best permalink structure for a site I'm toying with now. 1. I know that WordPress wants a unique number in the post to improve performance and db calls. 2. I know that for basic on-page SEO, most of us would opt for CATEGORY/POST or maybe even just post. I constantly change those. It's a bad habit, but sometimes you want the killer headline and a decent title in the post. So here is the issue: I can rewrite or use a plugin (anyone have a favorite) the permalinks to speed up site performance. We all know Google wants that. Maybe the permalink becomes /1234-foo But you know, a number in front of the URL just isn't awfully user friendly. If someone wants to read the foo post, it's nice to send them directly there. So would you trade off a slowdown in site speed for the prettiest permalinks for usability and SEO? And since you're asking a WP question, has anyone heard of a hard cap on static pages where the database starts dragging? The site I have in mind has 400 each posts and pages. Would moving platforms to Drupal or Joomla allow handling that many pages more effectively? Thanks for contributing and any help you can give. George
Technical SEO | | georgebounacos0