Robots.txt Help
-
I need help to create robots.txt file.
Please let me know what to add in the file. any real example or working example.?
-
Michael, from what i can tell, your website is built using WordPress. We typically recommend installing the Yoast SEO plugin and using that--which will help with your robots.txt file. If you need more information, take a look here: https://yoast.com/wordpress-robots-txt-example/
Generally, most of your site won't need to be disallowed in the robots.txt file, unless you're using tags and categories on your site. Yoast typically helps disallow the proper directories that you need to disallow.
One thing that you need to be aware of is the fact that you don't want to disallow your .CSS or .JS files on your site, many of the themes nowadays will put those files in your wp-admin folder--which by default typically gets disallowed.
-
This is the site I used to really get a good understanding of how to create a robots.txt file: http://www.robotstxt.org/
-
A very basic robots.txt file would look something like the below
User-agent: *
Sitemap: http://www.yourwebsite.com/sitemap.xml
Disallow: http://www.yourwebsite.com/url-you-dont-want-indexed
Disallow: http://www.yourwebsite.com/another-url-you-dont-want-indexedHope that helps
-
Include sitemaps. Disallow: Pages that you don't want indexed: search pages, login pages, core admin files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
#! (hashbang) check help needed
Does anybody have experience using hashbang? We tried to use it to solve indexation problem and I'm not fully sure do we use right solution now (developers did it with these FAQ and Guide to Ajax crawling as information source). One of our client has problem, that their e-shop categories, has solution where search engines aren't able to index all products. In this example a category, there is this "Näita kõiki (38)" that shows all category products for users but as I understand search engines aren't able to index it as /et#/activeTab=tab02 because of #. Now there is used #! (hashbang) and it is /et#!/activeTab=tab02. Is this correct solution? Also now example category URL is defferent for better indexation with:
Intermediate & Advanced SEO | | raido
/et#!/
../et And when tabs "TOP ja uued" and "Näita kõik" where activated/clicked then:
/et#/activeTab=tab01
/et#/activeTab=tab02 I tried to fetch it in Google Webmaster Tools but it seems it didn't work. I would appreciate it if anybody can check this solution?0 -
Help with 50 Plus Penalty, No Manual Action Reported
We have a client who had been ranking in Google’s top ten organic results for 2 of his major keywords last year. Currently Bing and Yahoo ranks his site #1 for both of these terms; the ranking pages that are appearing were specifically targeted with these words. As of now, the client appears to have a 50 plus penalty for these two keywords. Appearing #76 for one term and # 60 for the other. We were thinking of submitting a reconsideration request through Google Webmaster tools, but discovered that you aren’t allowed to do that unless a Manual Action has appeared, which in this case has not. The only problem we’ve had with the site from an SEO standpoint is that we recently discovered a website that had copied some of the product descriptions verbatim. The client contacted the site owner who took it down immediately (about a month ago), but we still have not seen any improvement in rankings for these keywords. Does anyone have any ideas on how to communicate this Google and get the suspected penalty lifted if a reconsideration request is apparently not available?
Intermediate & Advanced SEO | | roundabout0 -
Have you guys seen this yet: panguintool.com to help identify what hit the ranks
Have you guys seen this yet: panguintool.com Works with GA to show traffic superimposed with Panda/Penguin/other
Intermediate & Advanced SEO | | irvingw
updates to help discover what hit the ranks. Looks interesting, requests access to your GA account.0 -
Can Bundling Products Help eCommerce SEO?
We currently have over 13,000 products on our site. SeoMoz reports many duplicate pages, which are items that are very similar (different size, application, sku, etc.). Would it be prudent to create a bundled product that has one page, one description, a set of images and a table with add to cart buttons for all of the different products on that page? (called a bundled product in Magento). Then create 301 redirects from all of the individual pages and categories to the relevant new bundled product.
Intermediate & Advanced SEO | | iJeep0 -
1200 pages no followed and blocked by robots on my site. Is that normal?
Hi, I've got a bunch of notices saying almost 1200 pages are no-followed and blocked by robots. They appear to be comments and other random pages. Not the actual domain and static content pages. Still seems a little odd. The site is www.jobshadow.com. Any idea why I'd have all these notices? Thanks!
Intermediate & Advanced SEO | | astahl110 -
Help, really struggling with fixing mistakes post-Penguin
We had previously implemented a strategy of paying for lots of links and focusing on 3 or 4 keywords as our anchors, which used to REALLY work (I know, I know, bad black hat strategy - I have since learned my lesson). These keywords and others have since plummeted up to 100 spots since Panda 3.3 and Penguin. So I'm trying to go in and fix all our mistakes cuz our domain is too valuable to us just to start over from scratch. Yesterday I literally printed a 75 page document of all of our links according to Open Site Explorer. I have been going in and manually changing anchor text wherever I can, and taking down the very egregious links if possible.This has involved calling and emailing webmasters, digging up old accounts and passwords, and otherwise just trying to diversify our anchor text and remove bad links. I've also gone into our site and edited some internal links (also too weighty on certain keywords) and removed other links entirely. My rankings have gone DOWN more today. A lot. WTF does Google want? Is there something I'm doing wrong? Should we be deleted links from all private networks entirely or just trying to vary the anchor text? Any advice greatly appreciated. Thanks!
Intermediate & Advanced SEO | | LilyRay0 -
Anchor text help
Hello, I am a small website designer in Mexico.. as you know it is hard to rank in these keywords My main competitors appear with (diseño web = web design) on google.com.mx Almost 80% of the anchor text from my links are (diseño web Mexico =web design Mexico) If I search (diseño web Mexico =web design) on google.com.mx I appear on first page. If I search (diseño web = web design) on google.com.mx I appear on page 3. My questions: Is this because my anchor text is diseño web Mexico? should i change it to (diseño web = web design)? Or is it because (diseño web = web design) is a harder key word to rank? this is my website http://bit.ly/eKyWvr Regards
Intermediate & Advanced SEO | | Pixelar0 -
Robots.txt disallow subdomain
Hi all, I have a development subdomain, which gets copied to the live domain. Because I don't want this dev domain to get crawled, I'd like to implement a robots.txt for this domain only. The problem is that I don't want this robots.txt to disallow the live domain. Is there a way to create a robots.txt for this development subdomain only? Thanks in advance!
Intermediate & Advanced SEO | | Partouter0