Is there any value in having a blank robots.txt file?
-
I've read an audit where the writer recommended creating and uploading a blank robots.txt file, there was no current file in place. Is there any merit in having a blank robots.txt file?
What is the minimum you would include in a basic robots.txt file?
-
I know this is four years old, but there's value in having a blank robots.txt as some tools (including the latest version of the Moz crawler) will baulk at sites without a robots.txt file.
-
Thanks for both of your replies. As per my question it was around whether there is any value having a blank robots.txt file. Philipp's answer was right on the money.
-
i mentioned same only, The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site."
n has added - More and more people use robots,txt to disallow access to some administration or private folders of the site
-
No use in having a blank robots.txt. Minimum requirement if you want to have your site crawled is this:
User-agent: * Allow: /
Note that Gagans example above will block the entire site.
-
Hi, This is what i got
" Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called_The Robots Exclusion Protocol_. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:
User-agent: * Disallow: /
The "<tt>User-agent: *</tt>" means this section applies to all robots. The "<tt>Disallow: /</tt>" tells the robot that it should not visit any pages on the site."
More and more people use robots,txt to disallow access to some administration or private folders of the site . If you dont want to hide anything then may be you can leave it blank
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt in subfolders and hreflang issues
A client recently rolled out their UK business to the US. They decided to deploy with 2 WordPress installations: UK site - https://www.clientname.com/uk/ - robots.txt location: UK site - https://www.clientname.com/uk/robots.txt
Technical SEO | | lauralou82
US site - https://www.clientname.com/us/ - robots.txt location: UK site - https://www.clientname.com/us/robots.txt We've had various issues with /us/ pages being indexed in Google UK, and /uk/ pages being indexed in Google US. They have the following hreflang tags across all pages: We changed the x-default page to .com 2 weeks ago (we've tried both /uk/ and /us/ previously). Search Console says there are no hreflang tags at all. Additionally, we have a robots.txt file on each site which has a link to the corresponding sitemap files, but when viewing the robots.txt tester on Search Console, each property shows the robots.txt file for https://www.clientname.com only, even though when you actually navigate to this URL (https://www.clientname.com/robots.txt) you’ll get redirected to either https://www.clientname.com/uk/robots.txt or https://www.clientname.com/us/robots.txt depending on your location. Any suggestions how we can remove UK listings from Google US and vice versa?0 -
Target: blank. Does it make an SEO difference?
I've notice many sites MOZ included no longer use the target: blank attribute. I think that's what it's called. Basically when a link on your site opens a new tab in the browser as opposed to replacing the browser window you are in. Given that MOZ think of everything, I would love to hear opinions on this.
Technical SEO | | wearehappymedia0 -
Will an XML sitemap override a robots.txt
I have a client that has a robots.txt file that is blocking an entire subdomain, entirely by accident. Their original solution, not realizing the robots.txt error, was to submit an xml sitemap to get their pages indexed. I did not think this tactic would work, as the robots.txt would take precedent over the xmls sitemap. But it worked... I have no explanation as to how or why. Does anyone have an answer to this? or any experience with a website that has had a clear Disallow: / for months , that somehow has pages in the index?
Technical SEO | | KCBackofen0 -
Does iFrame content between the iFrame tag produce any conceivable value????
One of my clients is using a variety of unbounce pages for their web forms that are held in iFrames in the site template. This doesn't seem to produce much rank as it ranks very low in the search engines, and doesn't list in SEOMOZ as relevant to the keywords that I'm targetting it for. I'm thinking it may provide some benefit to replicate the content in between the iFrame tag, i.e. <iframe src="UnbouncePage.html">replicated content here</iframe>. Any thoughts on this Mozers?? Thankss
Technical SEO | | AndyfromVerb0 -
Getting home page content at top of what robots see
When I click on the text-only cache of nlpca(dot)com on the home page http://webcache.googleusercontent.com/search?q=cache:UIJER7OJFzYJ:www.nlpca.com/&hl=en&gl=us&strip=1 our H1 and body content are at the very bottom. How do we get the h1 and content at the top of what the robots see? Thanks!
Technical SEO | | BobGW0 -
What is the value of english links with foreign language anchor text for a foreign site?
I have a site in Spanish that is hosted in Spain with a .es TLD. I already have many Spanish-language links from websites in Spain, but I obviously want more and I'm finding I might need to look beyond typical Spanish sites. In talking to some of my link builders who work on my English/American sites, they are recommending that I build links on the normal article sites, blogs and web 2.0 sites that I normally build links on but that I make all the content English and insert the anchor text in Spanish. For example, if my site were about "weightloss", my keyword would be "perder peso" (in spanish). They are recommending that I have articles, reviews, etc written about weightloss in English with the anchor text "perder peso" worked into the English article. Most of the sites are English sites that are hosted in the US (article sites, web 2.0 properties, etc). My question is what is the value of these links? Does anybody have any experience with this?
Technical SEO | | jargomang0 -
Restricted by robots.txt and soft bounce issues (related).
In our web master tools we have 35K (ish) URLs that are restricted by robots.txt and as have 1200(ish) soft 404s. WE can't seem to figure out how to properly resolve these URLs so that they no longer show up this way. Our traffic from SEO has taken a major hit over the last 2 weeks because of this. Any help? Thanks, Libby
Technical SEO | | GristMarketing0 -
Toolbar sais 33 PA & DA, does Google have the same values?
Hey Mozzers, I've moved a domain 301 redirect to a new site... after around 2-3 weeks the seomoz toolbar sais i have PA & DA 33... which were both 1 as being a new domain. Does Google credit the same kind of value to the new website if seomoz found the PA & DA being "33"?
Technical SEO | | mosaicpro0