How to prevent directory from being accessed by search engines?
-
Pretty much as the question says, is there any way to stop search engines from crawling a directory? I am working on a Wordpress installation for my site but don't want it to be listed in search engines until it's ready to be shown to the world. I know the simplest way is to password-protect the directory but I had some issues when I tried to implement that so I'd like to see if there's a way to do it without passwords. Thanks in advance.
-
But don't forget to remove that Disallow out of Robots.txt when you go live - if you want those pages to be indexed (and also the Meta-robots noindex nofollow).
Otherwise you might be pulling your hair out trying to figure out why none of your pages are getting indexed in the SERPs.
-
You're absolutely right! I left that part out. Thanks
-
The robots.txt file does not guarantee that your pages will not show up in search results! Your best bet after password protection is adding a NoIndex meta tag to you page headers.
Google have openly said that they obey this tag (Matt Cutts).
-
Xee,
It always help, and it is very easy to implement. This function to show the path to the sitemap ir very good.
-
It's not required to have the ending slash. At least, it works for us without it.
-
As it is, my site is just phpBB3 forums (www.bearsfansonline.com); would a sitemap really help that much?
-
If you don't have an robot.txt file, you need to include some important stuff first.
First, do you have a sitemap.xlm for your website? If not, its very important and you should creat it at: http://www.xml-sitemaps.com/
Create a robot.txt file and include the follow:
User-agent: * allow: / disallow: /directoryname
Sitemap: http://www.yousite.com/sitemap.xmlWith this you will inform all robots where is your sitemap. You should read more about robots.txt in this great post: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts
-
shouldn't you put a slash at the end of the directory in the robots file?
you can create the robots file through the Google Webmaster Tools
-
I don't have a robots.txt file in my root. Do I just create a text file, put the above lines into it, and upload it to my root after changing the name?
-
I'm assuming you want all search engines blocked from this directory. If so, edit your robots.txt file to state the following. This will block all bots from accessing a folder/directory on your site
User-agent: *
Disallow: /directoryname
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Adding directories to robots nofollow cause pages to have Blocked Resources
In order to eliminate duplicate/missing title tag errors for a directory (and sub-directories) under www that contain our third-party chat scripts, I added the parent directory to the robots disallow list. We are now receiving a blocked resource error (in Webmaster Tools) on all of the pages that have a link to a javascript (for live chat) in the parent directory. My host is suggesting that the warning is only a notice and we can leave things as is without worrying about the page being de-ranked/penalized. I am wondering if this is true or if we should remove the one directory that contains the js from the robots file and find another way to resolve the duplicate title tags?
Technical SEO | | miamiman1000 -
How should we handle re-directory links? Should we remove these links?
We are currently cleaning up bad links that were purchased by a previous SEO agency. We have found links on anonym.to pages that redirect traffic to our site automatically. How should this be handled? Should we remove these links?
Technical SEO | | Lorne_Marr0 -
Do other search engines use meta keywords
Just want to know even those Google states it doesn't use them is there any benefit from using them for the other search engines?
Technical SEO | | ReSEOlve0 -
Is a canonical tag the best solution for multiple search listing pages in a site?
I have a site where dozens of page listings are showing in my report with a parameter showing the page number for the listings. Is the best solution to canonical these page listings back a core page (all-products)? Or, do I change my site configuration in Webmasters to ignore "page" parameters? What's the solution? Example URL 1- http://mydomain.com/products/all-products?page=84 Example URL 2- http://mydomain.com/products/all-products?page=85 Example URL 3- http://mydomain.com/products/all-products?page=86 Thanks in advance for your direction.
Technical SEO | | JoshKimber0 -
URL or sitemap submit to search engines?
Hello, I have just updated content at some URL site links, and I also added new URL content. Should I submit URL or re-create a sitemap then submit it to search engines? And please advise me some tools for submit them?
Technical SEO | | JohnHuynh0 -
Is placing content in sub directories better for SERP
Hi For small web sites with less than 6 pages Is there a benefit to structuring url paths using keyword rich sub directories compared to pages in the root of the site. for example: domainname.co.uk/keywordpagename.html or www.domainname.co.uk/keyword/keywordpagename.html which seems to have better rankings? thanks keyword
Technical SEO | | Bristolweb0 -
How to prevent duplicate content in archives?
My news site has a number of excerpts in the form of archives based on categories that is causing duplicate content problems. Here's an example with the nutrition archive. The articles here are already posts, so it creates the duplicate content. Should I nofollow/noindex this category page along with the rest and 2011,2012 archives etc (see archives here)? Thanks so much for any input!
Technical SEO | | naturalsociety0 -
Anchor text in Flash Discoverable by Search Engines?
What recommendations do you all have to make anchor text discoverable in flash? More importantly is it even possible and does it contribute to link juice?
Technical SEO | | sunfever0