I am trying to block robots from indexing parts of my site..
-
I have a few websites that I mocked up for clients to check out my work and get a feel for the style I produce but I don't want them indexed as they have lore ipsum place holder text and not really optimized... I am in the process of optimizing them but for the time being I would like to block them. Most of my warnings and errors on my seomoz dashboard are from these sites and I was going to upload the folioing to the robot.txt file but I want to make sure this is correct:
User-agent: *
Disallow: /salondemo/
Disallow: /salondemo3/
Disallow: /cafedemo/
Disallow: /portfolio1/
Disallow: /portfolio2/
Disallow: /portfolio3/
Disallow: /salondemo2/
is this all i need to do?
Thanks
Donny
-
Thanks
-
this is the correct approach when using the robots.txt method of blocking. Be aware however, that the only secure way to ensure 100% that such locations are not indexed is to put them behind a password protected gate-way. I always recommend to design agencies that there be a simple single log-in screen between the front end and design folders. This can be as complex as unique UIDs and Passwords for every client, or a single shared login, if all you want to do is bar search engines from seeing the content.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help Setting Up 301 Redirects from Coldfusion Site to Wordpress Site.
I have created a new website and need to redirect all of the previous pages to the new one. The old website was built in coldfusion and the new site is built in wordpress. One of the pages I'm trying to redirect is www.norriseal.com/products.cfm to http://norrisealwellmark.com/products/. This is what I have in my .htaccess file <ifmodule mod_rewrite.c="">Options +FollowSymlinks
Technical SEO | | MarketHubb
RewriteEngine On
RewriteBase /
Redirect 301 /products.cfm http://norrisealwellmark.com/products/</ifmodule> The result of this redirect is http://norrisealwellmark.com/products.cfm How do I prevent the .cfm from appending to the destination URL?1 -
Site indexed by Google, but (almost) never gets impressions
Hi there, I have a question that I wasn't able to give it a reasonable answer yet, so I'm going to trust on all of you. Basically a site has all its pages indexed by Google (I verified with site:sitename.com) and it also has great and unique content. All on-page grades are A with absolutely no negative factors at all. However its pages do not get impressions almost at all. Of course I didn't expect it to be on page 1 since it has been launched on Dec, 1st, but it looks like Google is ignoring (or giving it bad scores) for some reason. Only things that can contribute to that could be: domain privacy on the domain, redirect from the www to the subdomain we use (we did this because it will be a multi-language site, so we'll assign to each country a subdomain), recency (it has been put online on Dec 1st and the domain is just a couple of months old). Or maybe because we blocked crawlers for a few days before the launch? Exactly a few days before Dec 1st. What do you think? What could be the reason for that? Thanks guys!
Technical SEO | | ruggero0 -
Index problems, Part 2
Hi Guy's A few weeks ago i posted a question:
Technical SEO | | Happy-SEO
https://moz.com/community/q/index-problems After some good advice, we changed a few things: www.domain.com <<< NL version www.domain.com/fr/ <<<< French version (domain.com/nl/ 301 redirect to domain.com). So the SERPS for keyword ‘shutters’ went from #32 to #8...... for 2 day's.... and gone.... and not comming back anymore.... Did we missed something? Help is much appreciated, thanks 🙂3 -
Does my "spam" site affect my other sites on the same IP?
I have a link directory called Liberty Resource Directory. It's the main site on my dedicated IP, all my other sites are Addon domains on top of it. While exploring the new MOZ spam ranking I saw that LRD (Liberty Resource Directory) has a spam score of 9/17 and that Google penalizes 71% of sites with a similar score. Fair enough, thin content, bunch of follow links (there's over 2,000 links by now), no problem. That site isn't for Google, it's for me. Question, does that site (and linking to my own sites on it) negatively affect my other sites on the same IP? If so, by how much? Does a simple noindex fix that potential issues? Bonus: How does one go about going through hundreds of pages with thousands of links, built with raw, plain text HTML to change things to nofollow? =/
Technical SEO | | eglove0 -
Why is this site ranking better than me
Hi just used the compare tool to try and find out why a site is ranking better than me http://www.opensiteexplorer.org/comparisons?site=www.lifestylemonthly.co.uk%2F my site is www.in2town.co.uk and the site i am comparing with is http://www.lifestylemonthly.co.uk/ Can anyone explain what is going on and how i can achieve better ranking results
Technical SEO | | ClaireH-1848860 -
Redirect from old wordpress site to new php site? Best approach
Hi I have two websites one legacy site done in wordpress the other in php. However I would like to merge the two together and remove the wordpress site. However it has a good link profile and the pages rank well. What is the best approach to do a 301 redirect from the old site with all its pages pointing to the homepage of the new site? If so what's the best way to do this in wordpress? Many thanks
Technical SEO | | ocelot0 -
Do you get credit for an external link that points to a page that's being blocked by robots.txt
Hi folks, No one, including me seems to actually know what happens!? To repeat: If site A links to /home.html on site B and site B blocks /home.html in Robots.txt, does site B get credit for that link? Does the link pass PageRank? Will Google still crawl through it? Does the domain get some juice, but not the page? I know there's other ways of doing this properly, but it is interesting no?
Technical SEO | | DaveSottimano0 -
Site Disappeared off of Search
A friend of mine has a site (http://bit.ly/q4iWkM ) that was ranking number one for their key word (Drimnagh() and has now completely disappeared off of the ranking. I did some checking and can't see a problem. She does have duplicate meta and titles throughout but this shouldn't be a punishable offence that I know of and is something that I am going to correct with a quick plugin install. I couldn't see any redirects or code stopping search either. When you do site:URL it shows up OK as well. She is client of mine (for website not for SEO) and she is really upset about it so any help from the forum would be appreciated. This isn't even a site I did but you couldn't get a better person to work with so I am eager to help where and if possible. Guinness all round if someone solves it next time you are in Ireland
Technical SEO | | kdaly1000