Robots.txt file
-
Does it serve any purpose if we omit robots.txt file ? I wonder if spider has to read all the pages, why do we insert robots.txt file ?
-
As Ryan said, robots.txt file is very useful when you wanna block (disallow) some pages. Indeed, if you don't want that spider crawls your page you must use robots.txt (noindex tags will let bot crawls, but not index, your page). I have got a small website but i dropped robots.txt in my folder. Maybe write just Allow: / could be useless, but you can say: "I respect protocols"
-
A good source to learn about the robots.txt file is here: http://www.robotstxt.org/
The robots.txt file is completely optional. I don't use the file at all on small sites.
The file offers a means to block crawlers which choose to honor the file's instructions from crawling all or part of a site. It also provides the location of a sitemap.
To that end, sitemaps are completely unnecessary for SEO assuming your site has proper navigation. Even if you choose to use a sitemap, you can offer the location via WMT rather then the robots.txt file.
With respect to blocking areas of your site, the primary use would be for CMS, forums, ecommerce or other sites where the software was limited and does not allow the site owner to use noindex on all pages.
As a rule, robots.txt should simply never be used except as a means of last resort. In my experience the file is overused by site owners and SEOs. One exception where I use a robots.txt is during a site's development when I do not wish the site to be crawled at all.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
.htaccess file uploaded, website won't load
I uploaded the .htaccess file with the below, and now my website won't load at all? Then I deleted the htaccess file and it still won't load? But then it would load on my phone when I took it down, not on chrome, or explorer? Then I put it back up and looked again on my phone, wouldn't load on phone. Then deleted file and it still won't load on my phone? What is going on? RewriteEngine on
On-Page Optimization | | dwebb007
RewriteCond %{HTTP_HOST} !^http://freightetc.com$
RewriteRule ^(.)$ http://www.freightetc.com/$1 [R=301]
RewriteCond %{THE_REQUEST} ^./index.php
RewriteRule ^(.*)index.php$ http://www.freightetc.com/$1 [R=301]0 -
Is it better to put all your CSS in 1 file or is it no problem to use 10 files or more like on most frameworks?
Is it better to put all your CSS in 1 file or is it no problem to use 10 files or more like on most frameworks?
On-Page Optimization | | conversal0 -
Using Robots Meta Tag on Review Form Pages
I have gone over this so many times and I just can't seem to get it straight and hope someone can help me out with a couple of questions: Right now, on my dynamically created pages created by filters (located on the category pages) I am using rel""canonical" to point them to their respective category page. Should I also use the robots meta tag as well? Similarly, each product I have on my site has a review form on it and thus is getting indexed by Google. I have placed the same canonical tag on them as well pointing them to the page with the review form on it. In the past I used robots.txt to block google from the review pages but this didn't really do much. Should I be using the robots meta tag on these pages as well? If I used the robots meta tag should I noindex,nofollow? Thanks in advance, Jake
On-Page Optimization | | jake3720 -
Is it redundant to include a redirect to my canonical domain (www) in my .htaccess file since I already have the correct rel="canonical" in my header?
I've been reading the benefits of each practice, but not found anyone mentioning whether it's really necessary to do both? Personally I try to stay clear of .htaccess rewrites unless it's absolutely necessary, since because I've read they can slow down a website.
On-Page Optimization | | HOPdigital0 -
Duplication About PDF Files on Website
Hello, My site's URL (web address) is: http://www.vostastores.com/ Above is the Our Website URL. We are in the process of Upgrading Our Website and for that we are adding all Details of each and every products. One of the thing that we are planning to do is to get Manufacturer's product PDF files on our Website which the manufacturer already have on their website. So our Question is that Since the manufacturer has the file on their website and we want to add the same on our website, Will be there any Duplication issue? If yes, then please provide us with a Solution by which we can add the same on our website. Thanks & Regards.
On-Page Optimization | | CommercePundit0 -
How do we handle sitemaps in robots.txt when multiple domains point to same physical location?
we have www.mysite.net, www.mysite.se, www.mysite.fi and so on. all of these domains point to the same physical location on our webserver, and we replace texts given back to client depending on which domain he/she requested. My problem is this: How do i configure sitemaps in robots.txt when robots.txt is used by multiple domains? If I for instance put the rows Sitemap: http://www.mysite.net/sitemapNet.xml
On-Page Optimization | | nordicnetproducts
Sitemap: http://www.mysite.net/sitemapSe.xml in robots.txt, would that result in some cross submission error?0 -
Image file name, is it important
If I use the same image all over my site, do I need to change the file name to avoid duplicate? Different alt text will be use on those images
On-Page Optimization | | BigBlaze2050 -
Image files names - should each be different ?
Hi SEOs I have created a page on my website with a certain tutorial - let's say "How to change a tyre in your car". I have Grade A from On-page report using SEOmoz tool. I have put a relevant alt description for each photo I have put in that tutorial and I have used relevant keyword as a name for each image. So for example this would look like that: changing-tyre-1.jpg changing-tyre-2.jpg My question is should I name these file differently according to their content ? For example if on my photo there is some tool I have to use to change the tyre should the file have the name relevant to the subject of the article/tutorial on the page or to the content of the image ? Is that a stupid question ? Or am I getting too fussy ? Thanks
On-Page Optimization | | lolskizz0