Robots.txt | any SEO advantage to having one vs not having one?
-
Neither of my sites has a robots.txt file. I guess I have never been bothered by any particular bot enough to exclude it.
Is there any SEO advantage to having one anyways?
-
It's good practice, especially if you are operating a CMS that can create accessible URLs that cause duplicate content problems, create "junk" pages, etc. For example: http://www.asos.com/robots.txt
Google dislikes search results pages being indexed, so you can block those off, e.g. http://moz.com/robots.txt
You can disallow the archive.org bot if you don't want old versions of your site appearing in its search engine, and as others have said you can point to your xml sitemap.
It's not a bad resource to have at your disposal for site hygiene / maintenance reasons, but it's not an absolute necessity either.
-
There are actually a couple good reasons but in short, it's "best practice" so it won't hurt by adding it in. It wont take more than a couple minutes.
-
Just good practice. One SEO advantage would be to include a reference to your sitemap within the robots.txt file.
Aside from that, if you want all of your pages crawled and don't have a sitemap (although you should), no need for a robots.txt file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Clarification regarding robots.txt protocol
Hi,
Technical SEO | | nlogix
I have a website , and having 1000 above url and all the url already got indexed in Google . Now am going to stop all the available services in my website and removed all the landing pages from website. Now only home page available . So i need to remove all the indexed urls from Google . I have already used robots txt protocol for removing url. i guess it is not a good method for adding bulk amount of urls (nearly 1000) in robots.txt . So just wanted to know is there any other method for removing indexed urls.
Please advice.0 -
Google is indexing blocked content in robots.txt
Hi,Google is indexing some URLs that i don't want to be indexed and also is indexing the same URLs with https. This URLs are blocked in the file robots.txt.I've tried to block this URLs through Google WebmasterTools but Google doesn't let me do it because this URL are httpsThe file robots.txt is correct so, what can i do to avoid this content to be indexed?
Technical SEO | | elisainteractive0 -
SEO Issues
Hi, We have created a moving cost calculator tools and other moving company can added this tools their website. This is the code: [ <iframe src="http: www.enakliyat.com.tr="" fiyat-hesapla.aspx"="" height="554" width="400" frameborder="0" scrolling="no" style="border:none;"> ] when the other moving company added this code their websites, tool also works on the site and the tool make the referrals traffic our site.** Is it right using this method?**</iframe src="http:> http://www.enakliyat.com.tr/evden-eve-nakliyat-fiyatlari-hesaplama/ here is the tool
Technical SEO | | iskq0 -
Robots.txt
Google Webmaster Tools say our website's have low-quality pages, so we have created a robots.txt file and listed all URL’s that we want to remove from Google index. Is this enough for the solve problem?
Technical SEO | | iskq0 -
Hit by a penalty? But which one?
Hi guys, First off, thanks for reading my post! My site, www.doctorloanusa.com suffered a massive drop in Google rankings and traffic on September 27. For the past 4 years, I've ranked in spots 1-3. My main keywords are physician loans and doctor loans. I had a competitor copy almost ALL my text. In early October, I filed a DMCA takedown and had the infringing site removed from the SERPS. I also made text changes to my site. I thought this duplicate content was the problem, but I have yet to see any improvements 2 months later. My site isn't particularly deep content-wise, but it's competitive with others in my niche. I have relatively few links but others in the SERPS have very similar link profiles. I tried to clean up a few questionable links and used the disavow tool. I have used the SEOmoz on-page tools to grade my site. Typically, I receive grade A reports. I still rank for my website URL, and still rank 1 in both Yahoo and Bing. My bounce rate is 43%. If anyone could provide insight, I would be very grateful. I need to get this site ranking again, as this property is my main source of income. Is the site over-optimized? Is it my anchor text profile? Thank you very much in advance!
Technical SEO | | Cary_Forest0 -
Pager + SEO - Is it possible?
Hi, I am having this issue. I know that pager are not friends with SEO, but I want to know which is the best to do in this situations. for example, I work in a news company, and I have a lot of news pages that are very extensive so I use a pager. Well here I have the problem. suppose that the url is www.mysite.com/news/id/here-comes-the-title When you enter that URL you are viewing the first page that has this meta: title keywords description Now, the problem comes when the user goes to the page 2 of this news article. What I shall do? 1- Change the url to www.mysite.com/news/id/here-comes-the-title-PAGE2 www.mysite.com/news/PAGE2-id/here-comes-the-title www.mysite.com/news/id/PAGE2/here-comes-the-title 2- in the page 2,3,4,5 ... add a meta robot noindex? In the option 2 I think that I am loosing the opportunity to index the body of my article. Is this correct? Thanks
Technical SEO | | informatica8100 -
SEO Audit - Panda
I am looking for a reputable SEO company to help diagnose Panda issues. I am very familiar with SEO and lead an in-house team so I need more than a basic audit. e.g You need unique content
Technical SEO | | WEB-IRS
e.g. You need to create quality content I am looking for someone with a technical mind to help diagnose. Please reach if you have someone in mind.0 -
Mobile site: robots.txt best practices
If there are canonical tags pointing to the web version of each mobile page, what should a robots.txt file for a mobile site have?
Technical SEO | | bonnierSEO0