Robots.txt Showing in SERP Results
-
Currently doing a technical audit for a website and when I search "Site:website.com -www" the only result is website.com/robots.txt
I was wondering if anyone else has come across this before -- or what this may mean from a technical audit standpoint.
Thank you!
-
nonsense. Search for https://www.google.com.au/search?q=inurl%3Arobots.txt&pws=0
Some of the first results with visible robots.txt I see are:
I refuse to believe that "something is seriously wrong" with any of these sites.
-
It's quite common for Google to index robots.txt files. (and also, rather odd) But check out all of these robots.txt files:
https://www.google.com/search?q=inurl%3Arobots.txt&pws=0&gl=us
So it's nothing to be alarmed by. With your particular query. "Site:website.com -www" it only shows pages indexed without the "www" so this just says that all the indexed pages most likely begin with www. The exception, of course, is the robots.txt file.
The bigger question for me is, why does Google cache robots.txt files? Oh well.
-
The robots.txt file should not show up. Sounds like there is something seriously wrong.
-
Did you also search for site:www.webiste.com? Are they blocking the site?What's in the actual robots file?
Also, does this happen when you search for the site in Bing?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No index tag robots.txt
Hi Mozzers, A client's website has a lot of internal directories defined as /node/*. I already added the rule 'Disallow: /node/*' to the robots.txt file to prevents bots from crawling these pages. However, the pages are already indexed and appear in the search results. In an article of Deepcrawl, they say you can simply add the rule 'Noindex: /node/*' to the robots.txt file, but other sources claim the only way is to add a noindex directive in the meta robots tag of every page. Can someone tell me which is the best way to prevent these pages from getting indexed? Small note: there are more than 100 pages. Thanks!
Technical SEO | | WeAreDigital_BE
Jens0 -
Why has google stopped showing one domain and switching to showing another that points to the same website?
My client has a website here: http://www.savannahchiropractic.com/ The site ranked really well for searches for the term "savannah chiroprtactor". Sometime last week (April 21 mobilegeddon?) the site slipped in rankings, but reappeared under the domain name: "www.whitemarshchiropractic.com" Both URLs go to the exact same site. and I don't even think that the whitemarshchripractic.com site is a redirect. I'm not sure how it's setup, but it's definitely not a duplicate site. The ranking is still lower than the savannahchiropractic.com site, but now appears. Anyone have any insights on this?
Technical SEO | | aj6130 -
Robots.txt on refinements
In dealing with Panda do you think it is a good idea to put all refinements for category pages in the robots.txt file? We already have a lot as noindex, follow but I am wondering if it would be better to address from a crawl perspective as the pages are probably thin duplicate content to Google.
Technical SEO | | Gordian0 -
Competitor in 70% of results?
I'm trying to rank in the UK for a keyword where one of the competitors takes up around 70% of the top 100 positions. How is that possible? Is this google really giving the most relevant results to searchers? I can't think that a user wants to be presented with the same site on nearly all of the top 10 pages?
Technical SEO | | SamCUK0 -
Homepage not showing for searches
Hi Looking for a bit advice our client - www.financial-wise.co.uk We worked with this client on his old website www.mortgage-wise.co.uk we had him ranking for most local searches. The client then re-branded the full company and got another company in to do this, they did him new website www.financial-wise.co.uk, the company then launched the new domain with the old one still showing, some of the content was the same, including homepage. Anyway the issue im having now is, for certain searches im struggling to get them ranking again and for searches such as financial wise, its inner pages showing instead of the homepage? I have che checked and homepage is indexed, i have also re-written all the text on the homepage but still having some issue, its almost like the homepage has been penalised any help would be great?
Technical SEO | | rfksolutionsltd0 -
Explain this search result
Hi folks, I came across a strange search result. Search on Google Australia for "income portfolio". http://www.google.com.au/search?sourceid=chrome&ie=UTF-8&q=income+portfolio See the first result? It's a login page. How is that search result showing? And in position #1! Where is it getting its title and descriptions tags from? Does Google have a way to somehow see what is behind the login? Appreciate your thought.
Technical SEO | | scotennis0 -
/$1 URL Showing Up
Whenever I crawl my site with any kind of bot or a sitemap generator over my site. it comes up with /$1 version of my URLs. For example: It gives me hdiconference.com & hdiconference.com/$1 and hdiconference.com/purchases & hdiconference.com/purchases/$1 Then I get warnings saying that it's duplicate content. Here's the problem: I can't find these /$1 URLs anywhere. Even when I type them in, I get a 404 error. I don't know what they are, where they came from, and I can't find them when I scour my code. So, I'm trying to figure out where the crawlers are picking this up. Where are these things? If sitemap generators and other site crawlers are seeing them, I have to assume that Googlebot is seeing them as well. Any help? My developers are at a loss as well.
Technical SEO | | HDI0