Robots.txt | any SEO advantage to having one vs not having one?
-
Neither of my sites has a robots.txt file. I guess I have never been bothered by any particular bot enough to exclude it.
Is there any SEO advantage to having one anyways?
-
It's good practice, especially if you are operating a CMS that can create accessible URLs that cause duplicate content problems, create "junk" pages, etc. For example: http://www.asos.com/robots.txt
Google dislikes search results pages being indexed, so you can block those off, e.g. http://moz.com/robots.txt
You can disallow the archive.org bot if you don't want old versions of your site appearing in its search engine, and as others have said you can point to your xml sitemap.
It's not a bad resource to have at your disposal for site hygiene / maintenance reasons, but it's not an absolute necessity either.
-
There are actually a couple good reasons but in short, it's "best practice" so it won't hurt by adding it in. It wont take more than a couple minutes.
-
Just good practice. One SEO advantage would be to include a reference to your sitemap within the robots.txt file.
Aside from that, if you want all of your pages crawled and don't have a sitemap (although you should), no need for a robots.txt file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Role of Robots.txt and Search Console parameters settings
Hi, wondering if anyone can point me to resources or explain the difference between these two. If a site has url parameters disallowed in Robots.txt is it redundant to edit settings in Search Console parameters to anything other than "Let Googlebot Decide"?
Technical SEO | | LivDetrick0 -
How can I make it so that robots.txt is not ignored due to a URL re-direct?
Recently a site moved from blog.site.com to site.com/blog with an instruction like this one: /etc/httpd/conf.d/site_com.conf:94: ProxyPass /blog http://blog.site.com
Technical SEO | | rodelmo4
/etc/httpd/conf.d/site_com.conf:95: ProxyPassReverse /blog http://blog.site.com It's a Wordpress.org blog that was set as a subdomain, and now is being redirected to look like a directory. That said, the robots.txt file seems to be ignored by Google bot. There is a Disallow: /tag/ on that file to avoid "duplicate content" on the site. I have tried this before with other Wordpress subdomains and works like a charm, except for this time, in which the blog is rendered as a subdirectory. Any ideas why? Thanks!0 -
A/B testing entire website VS Seo issues
I'm familar with A/B testing variations of a page but I'd like to A/B test a new designs version of a e-commerce site. I´m wondering about the best way to test with SEO concerns... this is what I´ve in mind right now, any suggestion? Use parameters to make version B different from A version. Redirect 50% of the users with 302 ( or javascript would be a better way?) Use noindex in the B pages. Use rel=canonical in the B pages pointing to A version. In the end use 301 redirect to all B pages to A urls. PS: We can´t use subdomain and i don´t wanna use robots.txt file to protect the new design from competitors. I´d love any suggestions and tips about it - thanks folks 🙂
Technical SEO | | SeoMartin10 -
SEO problems from moving from several pages to one accordian
Ive read other posts that say using accordion is not detrimental to SEO, and for conversion optimization we want to take several of our existing pages and make them into one accordion. But what will this do to seo and duplicate content as I redirect the old pages to anchors in the accordion? I would think this would be a dup content problem as www.oldinfo1 www.oldinfo2 will now have their content on the same page but I will be redirecting them to www.newpage#oldinfo1 www.newpage#oldinfo2 Is there a way around duplicate content problems?
Technical SEO | | JohnBerger0 -
Dedicated ip helpful for seo
I read somewhere a while back that having a dedicated ip address was helpful for seo if this true or just another rumor? Also I read you should purchase your domain name for multiple yrs, what do you guys think?
Technical SEO | | TinaGammon0 -
Does Bing ignore robots txt files?
Bonjour from "Its a miracle is not raining" Wetherby Uk 🙂 Ok here goes... Why despite a robots text file excluding indexing to site http://lewispr.netconstruct-preview.co.uk/ is the site url being indexed in Bing bit not Google? Does bing ignore robots text files or is there something missing from http://lewispr.netconstruct-preview.co.uk/robots.txt I need to add to stop bing indexing a preview site as illustrated below. http://i216.photobucket.com/albums/cc53/zymurgy_bucket/preview-bing-indexed.jpg Any insights welcome 🙂
Technical SEO | | Nightwing0 -
Are affiliate programs good for seo?
We found this website https://shareasale.com/learnmore.cfm Actually is an affiliate site, where the users there would be in charge of driving traffic into our website, and then we pay them in some way, I just wanted to know if this can affect my seo efforts negativele? THanks
Technical SEO | | levalencia10 -
SEO Impact of IPv4 and iPv6?
Thought I'd see what the asking side of Q&A feels like 😉 We've been hearing for forever that the internet is running out of IP addresses, but I finally encountered the reality of it. I just realized that one of my sites is on a shared IP (hosted by Hosting.com, formerly HostMySite.com). My other sites with them included a unique IP, so I was surprised to discover this. They claim it's due to limitations on their IP allocations. Hosting.com doesn't have the option to buy a unique IP, but some other hosts do. I noticed, though, that many of them are using IPv6 for the new accounts. Has anyone had practical experience with having an IPv6 address and is there any impact on SEO?
Technical SEO | | Dr-Pete0