Robots.txt questions...
-
All,
My site is rather complicated, but I will try to break down my question as simply as possible.
I have a robots.txt document in the root level of my site to disallow robot access to /_system/, my CMS. This looks like this:
# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism**User-agent: ***
Disallow: /_system/I have another robots.txt file in another level down, which is my holiday database - www.mysite.com/holiday-database/ - this is to disallow access to /holiday-database/ControlPanel/, my database CMS. This looks like this:
**User-agent: ***
Disallow: /ControlPanel/Am I correct in thinking that this file must also be in the root level, and not in the /holiday-database/ level? If so, should my new robots.txt file look like this:
# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism**User-agent: ***
Disallow: /_system/
Disallow: /holiday-database/ControlPanel/Or, like this:
# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism**User-agent: ***
Disallow: /_system/
Disallow: /ControlPanel/Thanks in advance.
Matt
-
Good answer Yannick.
here are some resources:
http://www.free-seo-news.com/all-about-robots-txt.htm
http://www.robotstxt.org/robotstxt.html
Good luck
-
Cheers gents.
-
Like:
# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism**User-agent: ***
Disallow: /_system/
Disallow: /holiday-database/ControlPanel/Search engines typically only look in the root of your domain to find robots.txt and sitemap.xml files.
-
Hey Matt
The first of your options looks right and google and other engines look for the robots.txt file in the site root rather than for each directory.
If you had a reason for not wanting that info in the root robots.txt file you can always use the robots meta tag on the pages in a given directory.
Few useful links:
Robots.txt
http://www.google.com/support/webmasters/bin/answer.py?answer=156449&&hl=enRobots Meta Tag
http://www.google.com/support/webmasters/bin/answer.py?answer=93710Marcus
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt allows wp-admin/admin-ajax.php
Hello, Mozzers!
Technical SEO | | AndyKubrin
I noticed something peculiar in the robots.txt used by one of my clients: Allow: /wp-admin/admin-ajax.php What would be the purpose of allowing a search engine to crawl this file?
Is it OK? Should I do something about it?
Everything else on /wp-admin/ is disallowed.
Thanks in advance for your help.
-AK:2 -
2 sitemaps on my robots.txt?
Hi, I thought that I just could link one sitemap from my site's robots.txt but... I may be wrong. So, I need to confirm if this kind of implementation is right or wrong: robots.txt for Magento Community and Enterprise ...
Technical SEO | | Webicultors
Sitemap: http://www.mysite.es/media/sitemap/es.xml
Sitemap: http://www.mysite.pt/media/sitemap/pt.xml Thanks in advance,0 -
301 redirect question
Hi Everyone When doing 301 redirects for a large site, if a page has 0 inbound links would you still redirect it or just leave it? Im just curious on the best practice for this Thanks in advance
Technical SEO | | TheZenAgency0 -
Detailed ranking question for the pros
Hi Community, We've been struggling with the search engine ranking of our SEO optimised homepage for a number of months. I'm going to provide an overview of the page stats in hope that somebody might have a suggestion as to what the problem might be or where we should be focusing our efforts. I have also provided the stats of our main competitor as I have no idea why they are ranking so high based on the stats provided: URL in question: https://mysite.com On Page Grade for our targeted keyword: A Domain authority: 36 Page authority: 45 Root Doman Links: 57 Total Links: 634 SE Ranking: #17 Competitor URL in question: https://competitorsite.com On Page Grade for same targeted keyword: A Domain authority: 32 Page authority: 43 Root Doman Links: 28 Total Links: 919 SE Ranking: #1 Another strange this about our homepage is that a second tier page on our site is actually ranking higher in the search ranking for the Targeted Keyword (#9), even though this page has not been optimised and has an On Page Grade of F. Does anybody have any suggestions of what we might be overlooking or what the issue here might be? -JF
Technical SEO | | ERpro0 -
Robots.txt
I have a client who after designer added a robots.txt file has experience continual growth of urls blocked by robots,tx but now urls blocked (1700 aprox urls) has surpassed those indexed (1000). Surely that would mean all current urls are blocked (plus some extra mysterious ones). However pages still listing in Google and traffic being generated from organic search so doesnt look like this is the case apart from the rather alarming webmaster tools report any ideas whats going on here ? cheers dan
Technical SEO | | Dan-Lawrence0 -
Mobile or Responsive canonical question?
Hi guys We are in the process of expanding and are moving our site to magento enterprise. Today we met with a company pitching a seperate mobile site. While Im al for a mobile site in terms of look and user experience, from an seo point i dont believe and "m." domain is the best idea. However if we were to go with a mobile site, would adding canonical tags to the mobile urls pointing to the desktop urls be useful? For example m.trespass.co.uk/category-page has the canonical tag pointing to trespass.co.uk/category-page Im looking for someone who has direct experience wth this situation for one of their clients. Thanks Robert
Technical SEO | | Trespass0 -
Long Domain Name - Subpage URL Question
I have a long domain name, so domainname/services/page title can get pretty lengthy. I have a services page, as a summary page since there's a few of htem, with more detailed on the actual page. In this situation, would it be better to do domainname.com/services/service-name which can exceed the suggested 70 characters, or would it be a better idea to do domain.com/service-name and just have hte m under the services menu? Is there any advantage/disadvantage to going out 2-3 tiers? or having the sub pages of those services off the domain instead of a child of the root child page Please let me know if any clarification is needed. Thanks!
Technical SEO | | tgr0ss0