Robots.txt and robots meta
-
I have an odd situation. I have a CMS that has a global robots.txt which has the generic
User-Agent: *
Allow: /I also have one CMS site that needs to not be indexed ever. I've read in various pages (like http://www.jesterwebster.com/robots-txt-vs-meta-tag-which-has-precedence/22 ) that robots.txt always wins over meta, but I have also read that robots.txt indicates spiderability whereas meta can control indexation. I just want the site to not be indexed. Can I leave the robots.txt as is and still put NOINDEX in the robots meta?
-
I see. Have you considered putting it behind an htpasswd?
-
I can control it (it's a custom piece of software) but it's not as easy a fix as adding a meta to the template.
The main problem is we have a junk TLD we use to test some new ideas off the live server (lets clients give us feedback) but it gets spidered and indexed and starts ranking for client sites before they're ready to live in their own TLD. This means we have to compete against ourselves (even with a 301). There's nothing sensitive or it would live behind a password.
-
Do you need to control access to the site beyond the SERPS? I would not rely on robots.txt to shield any sensitive data.
For a breakdown of robots.txt and robots meta-tags checkout: http://www.robotstxt.org/robotstxt.html and http://www.searchtools.com/robots/robots-meta.html/, and for a great post on using these standards in SEO check out: http://www.seomoz.org/blog/serious-robotstxt-misuse-high-impact-solutions
I am also concerned that you are unable to control your robots.txt! If your CMS doesn't let you do that and overwrites it when you change it manually, you have some major control problems on your hands that you should remedy.
-
Blocking it at the robots.txt will not guarantee that your site will not appear at Google's index. I think you can use meta robots NOINDEX to guarantee that Google will not show your pages when someone try to Google it.
It is important to say that Googlebot and other spiders will continue to visit your page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
If you use canonicals do the meta descriptions need to be different?
For example, we have 3 different subsites with the same pages. We will put canonicals so they reference the main pages. Do the meta descriptions have to be different for each of the three pages? How does Google handle meta data when using canonicals?
Technical SEO | | Shirley.Fenlason0 -
Duplication in Meta Titles
Hi,
Technical SEO | | ChrisHolgate
In order to appease the Moz crawler we recently changed over 10,000 URL's in order to make our Meta Page Title less than 55 characters as it suggested. Unfortunately our rankings dropped dramatically pretty much overnight so I am getting the feeling that perhaps our titles are now just a little too concise and need elaborating on just a touch. Our competitors that rank well seem to use a small amount of keyword repetition. For example, whereas we may have:
Brother DCP-197C Inkjet Cartridges They will have:
Brother DCP-197C Inkjet Cartridges. Cheap Brother DCP-197C Ink. What are your opinions of the fact that: a) Their Title is over the 55 character figure that is suggested for displaying correctly in the SERPs.
b) The words Brother and DCP-197C are repeated in the title. The fact their title appears to be working better is almost enough to sway me but the competitors title just looks a little too spammy for me to make a sitewide change without asking some second opinions first. Cheers all!0 -
Different meta descriptions for same page
Hi Depending what terms I put into Google I am seeing a different meta description for exactly the same page. I have checked Umbraco CMS and everything seems in working order. Is there a reason this would be happening? Anyone else had trouble like this?
Technical SEO | | TheZenAgency0 -
Moving a site including meta data
If a website is moved to a new server, how can you ensure that all meta data moves with it?
Technical SEO | | jazavide0 -
Would you move the site to a different host or change packages at a significant expense in order to eliminate the meta refresh
When I began working with a site (http://www.visix.com) , I discovered a number of hosting constraints that hampered some SEO related changes I wanted to make. A year later, the site was teetering on the 1st page for a particular keyword of choice and when the Panda & Penguin updates happened, the site got passed by 3M & Amazon, both much bigger sites. (was #11, now #13) Now I'm thinking I should try and use the homepage to rank for keyword "digital signage software", where originally I was making progress with an inner page. Now I am revisting the homepage meta refresh and need to decide if it is enough of an issue to warrant a hosting change. http://www.visix.com has a meta-refresh "0" seconds to http://www.visix.com/index.aspx I know sites can rank well with these, although I don't know the level of handicap that it has. In an article here, http://www.seomoz.org/learn-seo/redirection there is a statement saying that a meta-refresh will not pass as much link juice as a 301 redirect. I have read about every opinion I can find, and would appreciate other's opinions on the matter. The host is Network Solutions and the hosting package does not allow 301 redirects, among other things. Would you move the site to a different host or change packages at a significant expense in order to eliminate the meta refresh or is it not a big deal on a well established site? Thanks very much for your feedback!
Technical SEO | | IntegralOCR30 -
What does it mean by 'blocked by Meta Robot'? How do I fix this?
When i get my crawl diagnostics, I am getting a blocked by Meta Robot, which means that my page is not being indexed in the search engines... obviously this is a major issue for organic traffic!!! What does it actually mean, and how can i fix it?
Technical SEO | | rolls1230 -
What is the sense of robots.txt?
Using robots.txt to prevent search engine from indexing the page is not a good idea. so what is the sense of robots.txt? just for attracting robots to crawl sitemap?
Technical SEO | | jallenyang0