Robots.txt error message in Google Webmaster from a later date than the page was cached, how is that?
-
I have error messages in Google Webmaster that state that Googlebot encountered errors while attempting to access the robots.txt. The last date that this was reported was on December 25, 2012 (Merry Christmas), but the last cache date was November 16, 2012 (http://webcache.googleusercontent.com/search?q=cache%3Awww.etundra.com/robots.txt&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a).
How could I get this error if the page hasn't been cached since November 16, 2012?
-
That's what our next move is. I'll let you all know what comes of it. Thanks for the response!
-
I've noticed several discrepancies in google's cache system. It seems many of their documents lag and are not updated immediately. It could just be errors in cross data population. If you really want to know the last time the Google bot visited your website then you will want to visit your server logs.
If your server logs don't show a visit from google on the 25th then we really do have to wonder. My guess is that the webmaster tools is reflective of the correct date. Either way, I'd check for the error they are reporting.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Home Page Title - Google Overriding my Title Tag
Hi, We have noticed our home page title tag has now been replaced by our brand name (by Google I'm assuming). We have also noted that the page is dropping off SERPS for our main keywords, I suspect they are related but ofc I cant be sure. I know the recent Google update has impacted titles but I wasn't sure if it would apply here. Has anyone any advice on this and/or having the same issue? We normally rank well for grass seed (UK search) https://thegrasspeople.com/ I also noticed some strange mark up in our source code which seems to have been left behind by Sketch - we are getting this removed. <title>Combined Shape</title> <desc>Created with Sketch.</desc> Chris
Intermediate & Advanced SEO | | Chris_Mc0 -
How do i beat my spammy competitors? they are ranking on page 1 of google!
my competitor is ranking on page 1 of google. he has 3.4 million backlinks and 1400 refering domains. he has aquired these backlinks from various websites, but he doesnot have links from his niche . how do i beat my competition with less backlinks because if i follo his technique, it would a takea lot of time and people to build backlinks one of my strategy is to get .edu links second strategy is to have 6000 word content and rank for really low competition keywords related to my website.( my competitors website has 1500 words content!) any other strategy you can suggest?
Intermediate & Advanced SEO | | calvinkj0 -
Implications of extending browser caching for Google?
I have been asked to leverage browser caching on a few scripts in our code. http://www.googletagmanager.com/gtm.js?id=GTM-KBQ7B5 (16 minutes 22 seconds) http://www.google.com/jsapi (1 hour) https://www.google-analytics.com/plugins/ua/linkid.js (1 hour) https://www.google-analytics.com/analytics.js (2 hours) https://www.youtube.com/iframe_api (expiration not specified) https://ssl.google-analytics.com/ga.js (2 hours) The number beside each link is the expiration for cache applied by the owners. I'm being asked to extend the time to 24 hours. Part of this task is to make sure doing this is a good idea. It would not be in our best interest to do something that would disrupt the collection of data. Some of what I'm seeing is recommending having a local copy which would mean missing updates from ga/gtm or call for the creation of a cron job to download any updates on a daily basis. Another concern is would caching these have a delay/disruption in collecting data? That's an unknown to me – may not be to you. There is also the concern that Google recommends not caching outside of their settings. Any help on this is much appreciated. Do you see any issues/risks/benefits/etc. to doing this from your perspective?
Intermediate & Advanced SEO | | chrisvogel0 -
Need help with Robots.txt
An eCommerce site built with Modx CMS. I found lots of auto generated duplicate page issue on that site. Now I need to disallow some pages from that category. Here is the actual product page url looks like
Intermediate & Advanced SEO | | Nahid
product_listing.php?cat=6857 And here is the auto generated url structure
product_listing.php?cat=6857&cPath=dropship&size=19 Can any one suggest how to disallow this specific category through robots.txt. I am not so familiar with Modx and this kind of link structure. Your help will be appreciated. Thanks1 -
Google Search Console - Indexed Pages
I am performing a site audit and looking at the "Index Status Report" in GSC. This shows a total of 17 URLs have been indexed. However when I look at the Sitemap report in GSC it shows 9,000 pages indexed. Also, when I perform a site: search on Google I get 24,000 results. Can anyone help me to explain these anomalies?
Intermediate & Advanced SEO | | richdan0 -
Newly designed page ranks in Google but then disappears - at a loss as to why.
Hi all, I wondered if you could help me at all please? We run a site called getinspired365.com (which is not optimised) and in the last 2 weeks have tried to optimise some new pages that we have added. For example, we have optimised this page - http://getinspired365.com/lifes-a-bit-like-mountaineering-never-look-down This page was added to Google's index via webmaster tools. When I then did a search for the full quote it came back 2nd in Google's search. If I did a search for half the quote (Life is a bit like mountaineering) it also ranked 2nd. We had another quote page that we'd optimised that displayed similar behaviour (it ranked 4th). But then for some reason when I now do the search it doesn't rank in the top 100 results. This, despite, an unoptimised "normal" page ranking 4th for a search such as: Thousands of geniuses live and die undiscovered. So our domain doesn't seem to be penalised as our "normal" pages are ranking. These pages aren't particularly well designed from an SEO standpoint. But our new pages - which are optimised - keep disappearing from Google, despite the fact they still show as indexed. I've rendered the pages and everything appears fine within Google Webmaster Tools. At a bit of a loss as to why they'd drop so significantly? A few pages I could understand but they've all but been removed. Any one seen this before, and any ideas what could be causing the issue? We have a different URL structure for our new pages in that we have the quote appear in the URL. All the content (bar the quote) that you see in the new pages are unique content that we've written ourselves. Could it be that we've over optimised and Google view these pages as spam? Many thanks in advance for all your help.
Intermediate & Advanced SEO | | MichaelWhyley0 -
Google Page Rank Dead?
Does PR still work? I have sites that have PR3 and get almost no traffic and sites that are PR1 and get thousands of uniques per month. My PR on my main sites haven't moved for about 7 years, even though we've grown significantly. I know lots of you are going to jump in with get the MOZ toolbar, which I already have done, and I agree, it's great ... But can anyone tell me about what's going on with Google PR? Is it still active? Or has Google abandoned? I noticed that the Google toolbar is not even available for Google Chrome. That should say something ... If you like this question, do me a favor, and give me a THUMBS UP!
Intermediate & Advanced SEO | | applesofgold2 -
Will Google Revisit a 403 Page
Hi, We've got some pretty strict anti-scraping logic in our website, and it seems we accidentally snared a Googlebot with it. About 100 URL requests were responded to with a 403 Forbidden error. The logic has since been updated, so this should not happen again. I was just wondering if/when Googlebot will come back and try those URLs again. They are linked from other pages on the site, and they are also in our sitemap. Thanks in advance for any assistance.
Intermediate & Advanced SEO | | dbuckles0