Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Google Unable to Access Robots.txt

Algorithm Updates

690

rhoadesjohn last edited by

We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.

The site has been up and active for well over a year now. What are my next steps?

I have included a screenshot of the top half of the file. See anything wrong?

D3H5tgE.png
1 Reply Last reply
Reply Quote 0
Modi last edited by

Would you mind sharing the solution - if not related to domain intricacies

Curious to know - only for knowledge purpose

Many thanks
1 Reply Last reply
Reply Quote 0
rhoadesjohn last edited by

Thanks Phillip. I think we're on the way to discovering the issue.

I certainly appreciate the feedback!
1 Reply Last reply
Reply Quote 0
zeepartner last edited by

I think you should allow general access before beginning to disallow various sections below:

Allow: /

Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Any recent updates from Google or community on sub domains vs sub directories?

Hi all, This has been a debate for years and I have noticed most of the SEOs suggest to go or switch to sub directories instead of sub domains. Still is this the same or any new updates from Google or SEO community? We have moved a sub domain to sub directory last year. The result was sub directory content started ranking good; but no change in website rankings. Because of moving sub domains to sub directories, will the linkjuice/PR of the website gets diluted as the number of pages increases which will takeaway soe authority? Thanks
Algorithm Updates | | vtmoz

0
When Google is going to penalise webpages with autoplaying videos?

Hi, One of the most disgusting experiences while browsing web pages is the auto playing anonymous and highly non relevant ad videos at some corner of the page which buffer mostly and automatically plays. There are familiar websites which been following this activity. I don't think Google has ever rolled out an update to penalise these. Any idea why? Thanks
Algorithm Updates | | vtmoz

0
Google's spell check recognize a keyword with volume

When the keyword "acls recertification" (an important keyword for our client) is typed into the Google search box, the word "recertification" is underlined in red. Note that you only need to type "acls rec" to make the red underline appear.BUT, Google does not underline the word "recertification" when it is typed into the search box alone, nor does Google underline the word "recertification" when the following keywords are searched: cpr recertification bls recertification pals recertification ^These are all closely related to the keyword "acls recertification," so this spell check behavior is very inconsistent.Why does this matter? Because no matter how close you come to typing "acls recertification," Google's autocomplete suggestions never include "acls recertification" (because of the perceived misspelling?).BUT, Google does suggest "acls recertification online" in the dropdown menu. If you select the "acls recertification online" suggestion then backspace until the word "online" is gone, the red underline disappears, and "acls recertification" becomes an autocomplete suggestion. VERY strange behavior...I have replicated this issue on various depersonalized browsers and devices, so I am confident that this is not related to my personal settings.This keyword contributes to a large portion of our client's business (they specialize in acls certification and recertification), so you can imagine how concerning this is for us. Note that until very recently (3-4 months ago), this keyword did NOT have any spell-check issues. This keyword averages 2400 searches per month according to AdWords which should be enough volume to allow Google to recognize the correct spellingI posted this issue in the Google product forums, where I was advised to submit feedback directly on the search results page via Google's "feedback" link. I have submitted this feedback to Google, but I thought I would bring this to the MOZ community as well to see if anyone has experienced a similar issue, or has any ideas as to what could be causing this issue.
Algorithm Updates | | RyanKent

0
Would Google Remove Pages for Inactivity?

Hi, I've been watching the Total Indexed number for 4 domains that I work with for the last few months. In Google Webmaster Tools three of them were holding steady up until August-September, when suddenly they started declining by hundreds of thousands of URLs a week. I've asked my IT department and they say they haven't done anything technically different in the last few months that would affect indexation. I've also searched on google and on search marketing blogs to see if anyone else has experience this to no avail. As you can see in the image, the "Not Selected" pages have not increased so it appears this is not due to duplicate content (of which we have a lot). However, the "Ever Crawled" number is increasing. The only reasonable answer that I can conclude is that Google is now de-indexing inactive URLs? Anyone have a better answer? yIYDm.jpg
Algorithm Updates | | OfficeFurn

0
Proper Way To Submit A Reconsideration Request To Google

Hello, In previous posts, I was speaking about how we were penalized by Google for unnatural links. Basically 50,000 our of our 58,000 links were coming from 4-5 sites with the same exact anchor text and img alt tags. This obviously was causing our issues. Needless to say, I wen through the complete link profile to determine that all of the links besides this were of natrural origins. My question here is what is the accepted protocol of submitting a reinclusion request; For example, how long should it be? Should I disclose that I was in fact using paid links, and now that I removed (or at least nofollowed) them? I want to make sure that the request as good as it should so I can get our rankings up in a timely manner. Also, how long until the request is typically aknowledged? Thanks
Algorithm Updates | | BestOdds

0
Meta Title Not Showing up in Google

Hello Friends, I have a website, www.bollywoodshaadis.com. On 1st may we changed our servers and revamped our website as per SEO updated guidelines. For some strange reason Google is not showing site Meta Title when you search the website on Google. All it shows is the domain name in the meta title. However, when you search info:www.bollywoodshaadis.com it shows the right Meta tags. Any reason for this happening? I have never seen this before. Thank you in advance.
Algorithm Updates | | SEOcandy

0
Trying to figure out why one of my popular pages was de-indexed from Google.

I wanted to share this with everyone for two reasons. 1. To try to figure out why this happened, and 2 Let everyone be aware of this so you can check some of your pages if needed. Someone on Facebook asked me a question that I knew I had answered in this post. I couldn't remember what the url was, so I googled some of the terms I knew was in the page, and the page didn't show up. I did some more searches and found out that the entire page was missing from Google. This page has a good number of shares, comments, Facebook likes, etc (ie: social signals) and there is certainly no black / gray hat techniques being used on my site. This page received a decent amount of organic traffic as well. I'm not sure when the page was de-indexed, and wouldn't have even known if I had't tried to search for it via google; which makes me concerned that perhaps other pages are being de-indexed. It also concerns me that I have done something wrong (without knowing) and perhaps other pages on my site are going to be penalized as well. Does anyone have any idea why this page would be de-indexed? It sure seems like all the signals are there to show Google this page is unique and valuable. Interested to hear some of your thoughts on this. Thanks
Algorithm Updates | | NoahsDad

0
Today all of our internal pages all but completely disappeared from google search results. Many of them, which had been optimized for specific keywords, had high rankings. Did google change something?

We had optimized internal pages, targeting specific geographic markets. The pages used the keywords in the url title, the h1 tag, and within the content. They scored well using the SEOmoz tool and were increasing in rank every week. Then all of a sudden today, they disappeared. We had added a few links from textlink.com to test them out, but that's about the only change we made. The pages had a dynamic url, "?page=" that we were about to redirect to a static url but hadn't done it yet. The static url was redirecting to the dynamic url. Does anyone have any idea what happened? Thanks!
Algorithm Updates | | h3counsel

0