Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Any recent updates from Google or community on sub domains vs sub directories?
Hi all, This has been a debate for years and I have noticed most of the SEOs suggest to go or switch to sub directories instead of sub domains. Still is this the same or any new updates from Google or SEO community? We have moved a sub domain to sub directory last year. The result was sub directory content started ranking good; but no change in website rankings. Because of moving sub domains to sub directories, will the linkjuice/PR of the website gets diluted as the number of pages increases which will takeaway soe authority? Thanks
Algorithm Updates | | vtmoz0 -
SEO - Google Local Listing & Same Day Delivery
Hi We are looking to offer same day delivery if you're in a 20 mile radius to us. I'm trying to do some research on how to optimise this for Google organic listings. Would this be the same as optimising for a local business listing? I'm not sure where to start. Thanks! Becky
Algorithm Updates | | BeckyKey0 -
January 10, 2017 - Intrusive interstitials Google Update
Hi all, As everyone is most likely aware, Google have recently announced that if a site has intrusive intersitals that push the main content below the fold, will be downgraded in the SERP's from January 10th. At the moment we have a range of international sites, .ca, .com.au, .co.uk, .fr etc - if a user from a UK IP goes to a .ca site - a country switcher dialog will appear. I am aware that this may affect our sites performance in mobile search when the update comes out - however, if we block Google from seeing this - will they still pick it up? Thanks.
Algorithm Updates | | Brett-S0 -
Google's spell check recognize a keyword with volume
When the keyword "acls recertification" (an important keyword for our client) is typed into the Google search box, the word "recertification" is underlined in red. Note that you only need to type "acls rec" to make the red underline appear.BUT, Google does not underline the word "recertification" when it is typed into the search box alone, nor does Google underline the word "recertification" when the following keywords are searched: cpr recertification bls recertification pals recertification ^These are all closely related to the keyword "acls recertification," so this spell check behavior is very inconsistent.Why does this matter? Because no matter how close you come to typing "acls recertification," Google's autocomplete suggestions never include "acls recertification" (because of the perceived misspelling?).BUT, Google does suggest "acls recertification online" in the dropdown menu. If you select the "acls recertification online" suggestion then backspace until the word "online" is gone, the red underline disappears, and "acls recertification" becomes an autocomplete suggestion. VERY strange behavior...I have replicated this issue on various depersonalized browsers and devices, so I am confident that this is not related to my personal settings.This keyword contributes to a large portion of our client's business (they specialize in acls certification and recertification), so you can imagine how concerning this is for us. Note that until very recently (3-4 months ago), this keyword did NOT have any spell-check issues. This keyword averages 2400 searches per month according to AdWords which should be enough volume to allow Google to recognize the correct spellingI posted this issue in the Google product forums, where I was advised to submit feedback directly on the search results page via Google's "feedback" link. I have submitted this feedback to Google, but I thought I would bring this to the MOZ community as well to see if anyone has experienced a similar issue, or has any ideas as to what could be causing this issue.
Algorithm Updates | | RyanKent0 -
Google Rankings Dropped in Past Few Weeks
Hi All, I work for an online appliance retailer and over the past weeks, we've seen a drop in our google SERPs. This time last year we were ranking in the top 3 for our top converting key terms, but now we are ranking towards the bottom of the first page or even on the top of the second page with the big box stores now dominating for our key terms. Needless to say traffic for these pages has dropped off considerably. We still have quite a bit of traffic coming in for other key terms, but they don't convert as well. Is anyone else seeing the same thing? If so what are you doing to combat this? Do you have any suggestions? Thank you!
Algorithm Updates | | airnwater0 -
New Zealand rankings - Is Google taking city into account now?
I've just noticed over the last 24 hours that Google New Zealand seems to be taking city into account with the search results - when I search from Christchurch - I get CHCH companies and Auckland results when searching from Auckland. I would like to rank my client nationally - any tips? Has anyone else noticed this? I need to be the top of Google for my keyword throughout the country. Any articles that could help? Kind regards, David
Algorithm Updates | | David_Buckingham0 -
Google Local Algorithm Changes?
I was wondering if you have heard about any Google Local algorithm changes. We have about 200 franchise locations. Some of our locations have dropped significantly over the past few weeks. Locations that were showing up in the 1-3 positions are now no longer showing on the first page. This is for very relevant phrases for our main line of business (which is also in our business name)... ‘Phrase, CITY NAME’. These locations have plenty of positive Google reviews. We would typically rank well for a phrase like that based on our relevance. I did some brainstorming. Do you think any of these could have any impact? Google is all about things looking and feeling natural including link building, etc. We have used Yext which made a lot of changes across the web to fix addresses, etc. Do you think Google may be seeing this as unnatural? Too many changes at to many sites in to short a period of time? Along those same lines, do you think Google may be penalizing some of our franchise pages for being to ‘perfect’? It would be ‘natural’ for addresses to have some difference across the web and a bit unnatural to have them all match so perfectly. I know that Google has always stated the business name should be listed in Google Local the way it is listed to the general public. Things such as “Business Name Boston” should be listed as “Business Name”. Each of our franchise locations is named in house to reflect their geo location..... "Business Name Boston", "Business Name St. Louis". Many of our competitors also use the practice of attaching geo terms as well. Do you think we may be getting hit with a penalty now even though we have listed things on Google with the Geo term for years.... and is how WE refer to each location? Is it possible that by working with Yext, we drew attention to this practice? Should we remove our local listings geo term on Google Local? How about across the web? We are in a business that does not require customers to come to our location. Some of our locations have not suppressed the address in their local listings while others have. Many of our competitors have not. Do you think this could play into it? Some of our locations that are not showing in Local have good organic results. Have you heard anything about Google dropping Local if they show in organic? I know Google has been looking at social media more and more and I believe they will continue to do so. If our local pages have no social presence, could this adversely affect things? (I think this is probably not the case…. but wanted to throw it out there) I have noticed that in some cases where Local has dropped, we have multiple offices in that metro area. Is it possible that this could affect things? Have you heard of any Local algorithm changes? I know they are releasing a new dashboard sporadically, could this be in conjunction with a larger Local algorithm change? Our CMS tool does not allow us to change Title/Meta per page (I know... terrible!!). So every page has the same title and same meta description. (We are changing our CMS system! Can't wait!). Could this play into it? Thanks for any feedback!
Algorithm Updates | | MABES1 -
Stop google indexing CDN pages
Just when I thought I'd seen it all, google hits me with another nasty surprise! I have a CDN to deliver images, js and css to visitors around the world. I have no links to static HTML pages on the site, as far as I can tell, but someone else may have - perhaps a scraper site? Google has decided the static pages they were able to access through the CDN have more value than my real pages, and they seem to be slowly replacing my pages in the index with the static pages. Anyone got an idea on how to stop that? Obviously, I have no access to the static area, because it is in the CDN, so there is no way I know of that I can have a robots file there. It could be that I have to trash the CDN and change it to only allow the image directory, and maybe set up a separate CDN subdomain for content that only contains the JS and CSS? Have you seen this problem and beat it? (Of course the next thing is Roger might look at google results and start crawling them too, LOL) P.S. The reason I am not asking this question in the google forums is that others have asked this question many times and nobody at google has bothered to answer, over the past 5 months, and nobody who did try, gave an answer that was remotely useful. So I'm not really hopeful of anyone here having a solution either, but I expect this is my best bet because you guys are always willing to try.
Algorithm Updates | | loopyal0