Robots.txt Showing in SERP Results
-
Currently doing a technical audit for a website and when I search "Site:website.com -www" the only result is website.com/robots.txt
I was wondering if anyone else has come across this before -- or what this may mean from a technical audit standpoint.
Thank you!
-
nonsense. Search for https://www.google.com.au/search?q=inurl%3Arobots.txt&pws=0
Some of the first results with visible robots.txt I see are:
I refuse to believe that "something is seriously wrong" with any of these sites.
-
It's quite common for Google to index robots.txt files. (and also, rather odd) But check out all of these robots.txt files:
https://www.google.com/search?q=inurl%3Arobots.txt&pws=0&gl=us
So it's nothing to be alarmed by. With your particular query. "Site:website.com -www" it only shows pages indexed without the "www" so this just says that all the indexed pages most likely begin with www. The exception, of course, is the robots.txt file.
The bigger question for me is, why does Google cache robots.txt files? Oh well.
-
The robots.txt file should not show up. Sounds like there is something seriously wrong.
-
Did you also search for site:www.webiste.com? Are they blocking the site?What's in the actual robots file?
Also, does this happen when you search for the site in Bing?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SiteName Attribute Showing in Different Language in SERP
We are currently experiencing issues with our subdomain SiteName. Our parent company root domain is a Japanese language site, but we have an English subdomain that is for the United States primarily, and nearly rest of world for organic traffic. Our issue is that we have followed the guidelines here: https://developers.google.com/search/docs/appearance/site-names There was a large post on here with many responses including Googlers with issues others were having, but it has since been removed. Here is the code in place on our homepage: <script
Technical SEO | | Evan_Wright
type="application/ld+json"> { "@context": "https://schema.org",
"@type": "WebSite",
"name": "Mescius Developer Tools",
"alternateName": ["Mescius, inc.", "developer.mescius.com"],
"url": "https://developer.mescius.com" }
</script> Unfortunately this is what is appearing in the SERP. It is using the Japanese equivalent of our parent company. Screenshot 2024-02-23 at 3.37.55 PM.png Even though the relationship between root and subdomain should not be causing this, it seems like something is impacting this incorrect SiteName, and it is impacting CTR for the subdomain. Has anyone else experienced this and found a fix?0 -
Have I constructed my robots.txt file correctly for sitemap autodiscovery?
Hi, Here is my sitemap: User-agent: * Sitemap: http://www.bedsite.co.uk/sitemaps/sitemap.xml Directories Disallow: /sendfriend/
Technical SEO | | Bedsite
Disallow: /catalog/product_compare/
Disallow: /media/catalog/product/cache/
Disallow: /checkout/
Disallow: /categories/
Disallow: /blog/index.php/
Disallow: /catalogsearch/result/index/
Disallow: /links.html I'm using Magento and want to make sure I have constructed my robots.txt file correctly with the sitemap autodiscovery? thanks,0 -
Results Pages Duplication - What to do?
Hi all, I run a large, well established hotel site which fills a specific niche. Last February we went through a redesign which implemented pagination and lots of PHP / SQL wizzardy. This has left us, however, with a bit of a duplication problem which I'll try my best to explain! Imagine Hotel 1 has a pool, as well as a hot tub. This means that Hotel 1 will be in the search results of both 'Hotels with Pools' and 'Hotels with Hot Tubs', with exactly the same copy, affiliate link and thumbnail picture in the search results. Now imagine this issue occurring hundreds of times across the site and you have our problem, especially since this is a Panda-hit site. We've tried to keep any duplicate content away from our landing pages with some success but it's just all those pesky PHP paginated pages which doing us in (e.g. Hotels/Page-2/?classifications[]263=73491&classifcations[]742=24742 and so on) I'm thinking that we should either a) completely noindex all of the PHP search results or b) move us over to a Javascript platform. Which would you guys recommend? Or is there another solution which I'm overlooking? Any help most appreciated!
Technical SEO | | dooberry0 -
Homepage disappeared from Google Serp
I redirected my domain using this code in .htaccess : RewriteCond %{HTTP_HOST} ^xxxx.com
Technical SEO | | digitalkiddie
RewriteRule (.*) http://www.xxxx.com/$1 [R=301,L]
<ifmodule mod_rewrite.c="">RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L] RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]/)index.(html?|php)(?[^\ ])?\ HTTP/
RewriteRule ^(([^/]/)*)index.(html?|php)$ http://www.xxxx.com/$1 [R=301,L]</ifmodule> A day after I did it, got an error in GWMT "Google can't find your site's robots.txt" and my homepage disappeared from the result pages. When I try to open Google cache of the homepage I got an error 404. I generated new robots.txt, uploaded it , now the error doesnt show but still my homepage is not in the serps. Its been 3 days. What should I do ? Thanks in advance "Google can't find your site's robots.txt" error? - Pro ...0 -
Competitor in 70% of results?
I'm trying to rank in the UK for a keyword where one of the competitors takes up around 70% of the top 100 positions. How is that possible? Is this google really giving the most relevant results to searchers? I can't think that a user wants to be presented with the same site on nearly all of the top 10 pages?
Technical SEO | | SamCUK0 -
Have my SERP listings been hacked?
When you Google my site the organic search results look normal. The preview site images even display my actual site when you roll over the results. However, when you click a result you are directed to various Spam pages, not my website. How is this possible? This only happens when you click through from search engines. If you type the URL directly in your browser, you are not redirected to a spam site. the site: funeralhomeoptions.com Have any of you seen or experienced this before?
Technical SEO | | emmyjo0 -
Robots.txt Sitemap with Relative Path
Hi Everyone, In robots.txt, can the sitemap be indicated with a relative path? I'm trying to roll out a robots file to ~200 websites, and they all have the same relative path for a sitemap but each is hosted on its own domain. Basically I'm trying to avoid needing to create 200 different robots.txt files just to change the domain. If I do need to do that, though, is there an easier way than just trudging through it?
Technical SEO | | MRCSearch0 -
How does google know a search result is a search result?
In the google webmaster forums, google specifically states that you should not include search results in the google index. What is the best way to make dynamic, great content show in search results without receiving a penalty?
Technical SEO | | nicole.healthline0