Robots.txt Showing in SERP Results
-
Currently doing a technical audit for a website and when I search "Site:website.com -www" the only result is website.com/robots.txt
I was wondering if anyone else has come across this before -- or what this may mean from a technical audit standpoint.
Thank you!
-
nonsense. Search for https://www.google.com.au/search?q=inurl%3Arobots.txt&pws=0
Some of the first results with visible robots.txt I see are:
I refuse to believe that "something is seriously wrong" with any of these sites.
-
It's quite common for Google to index robots.txt files. (and also, rather odd) But check out all of these robots.txt files:
https://www.google.com/search?q=inurl%3Arobots.txt&pws=0&gl=us
So it's nothing to be alarmed by. With your particular query. "Site:website.com -www" it only shows pages indexed without the "www" so this just says that all the indexed pages most likely begin with www. The exception, of course, is the robots.txt file.
The bigger question for me is, why does Google cache robots.txt files? Oh well.
-
The robots.txt file should not show up. Sounds like there is something seriously wrong.
-
Did you also search for site:www.webiste.com? Are they blocking the site?What's in the actual robots file?
Also, does this happen when you search for the site in Bing?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Role of Robots.txt and Search Console parameters settings
Hi, wondering if anyone can point me to resources or explain the difference between these two. If a site has url parameters disallowed in Robots.txt is it redundant to edit settings in Search Console parameters to anything other than "Let Googlebot Decide"?
Technical SEO | | LivDetrick0 -
Horizontal Scroll in SERPs
Hello,
Technical SEO | | Koki.Mourao
I would love to find out how this is implemented. I do not know this function's name but it basically is a horizontal scroll which loads on top of a SERP. What is it called? How can I implement this. Is it special tags? What? How? Any tips would be very helpful. Thanks qJMLCCF0 -
Can Mobile Results show on Desktop results?
Hi everyone, I have a question - I have a client who has told me they had a problem with duplicate mobile pages showing on desktop results in place of their main URL pages They have a mobile domain mobile.domain.com. My understanding is that Google can differentiate mobile and nonmobile pages and hence you cannot see mobile results on desktop and duplication of pages is not a problem. Has anyone had this problem before? Is this possible? Thanks
Technical SEO | | CayenneRed890 -
Why has google stopped showing one domain and switching to showing another that points to the same website?
My client has a website here: http://www.savannahchiropractic.com/ The site ranked really well for searches for the term "savannah chiroprtactor". Sometime last week (April 21 mobilegeddon?) the site slipped in rankings, but reappeared under the domain name: "www.whitemarshchiropractic.com" Both URLs go to the exact same site. and I don't even think that the whitemarshchripractic.com site is a redirect. I'm not sure how it's setup, but it's definitely not a duplicate site. The ranking is still lower than the savannahchiropractic.com site, but now appears. Anyone have any insights on this?
Technical SEO | | aj6130 -
Blocked jquery in Robots.txt, Any SEO impact?
I've heard that Google is now indexing links and stuff available in javascript and jquery. My webmastertools is showing that some links are blocked in robots.txt of jquery. Sorry I'm not a developer or designer. I want to know is there any impact of this on my SEO? and also how can I unblock it for the robots? Check this screenshot: http://i.imgur.com/3VDWikC.png
Technical SEO | | hammadrafique0 -
Warnings for blocked by blocked by meta-robots/meta robots Nofollow...how to resolve?
Hello, I see hundreds of notices for blocked by meta-robots/meta robots nofollow and it appears it is linked to the comments on my site which I assume I would not want to be crawled. Is this the case and these notices are actually a positive thing? Please advise how to clear them up if these notices can be potentially harmful for my SEO. Thanks, Talia
Technical SEO | | M80Marketing0 -
Authorship and picture in search results
Tim and Kris Hallbom at this site: nlpca(dot)com are authors of several books and many great articles, and they would like their picture to show up in their search results. Articlebase.com contacted us and called it authorship, and said that they could get our picture to begin showing up in appropriate google searches. But we don't want to go through Articlebase.com, how do we do this? Thank you.
Technical SEO | | BobGW0 -
UK Serps
Hi there, I have a website that is aimed primarily at the US market. However, I have the option to make some content for people in my niche for people in the UK. Is there anything I can do to help the site rank on google.co.uk for the UK content, baring in mind that most of my content is aimed at the US. Thanks, Peter
Technical SEO | | PeterM220