How to prevent directory from being accessed by search engines?
-
Pretty much as the question says, is there any way to stop search engines from crawling a directory? I am working on a Wordpress installation for my site but don't want it to be listed in search engines until it's ready to be shown to the world. I know the simplest way is to password-protect the directory but I had some issues when I tried to implement that so I'd like to see if there's a way to do it without passwords. Thanks in advance.
-
But don't forget to remove that Disallow out of Robots.txt when you go live - if you want those pages to be indexed (and also the Meta-robots noindex nofollow).
Otherwise you might be pulling your hair out trying to figure out why none of your pages are getting indexed in the SERPs.
-
You're absolutely right! I left that part out. Thanks
-
The robots.txt file does not guarantee that your pages will not show up in search results! Your best bet after password protection is adding a NoIndex meta tag to you page headers.
Google have openly said that they obey this tag (Matt Cutts).
-
Xee,
It always help, and it is very easy to implement. This function to show the path to the sitemap ir very good.
-
It's not required to have the ending slash. At least, it works for us without it.
-
As it is, my site is just phpBB3 forums (www.bearsfansonline.com); would a sitemap really help that much?
-
If you don't have an robot.txt file, you need to include some important stuff first.
First, do you have a sitemap.xlm for your website? If not, its very important and you should creat it at: http://www.xml-sitemaps.com/
Create a robot.txt file and include the follow:
User-agent: * allow: / disallow: /directoryname
Sitemap: http://www.yousite.com/sitemap.xmlWith this you will inform all robots where is your sitemap. You should read more about robots.txt in this great post: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts
-
shouldn't you put a slash at the end of the directory in the robots file?
you can create the robots file through the Google Webmaster Tools
-
I don't have a robots.txt file in my root. Do I just create a text file, put the above lines into it, and upload it to my root after changing the name?
-
I'm assuming you want all search engines blocked from this directory. If so, edit your robots.txt file to state the following. This will block all bots from accessing a folder/directory on your site
User-agent: *
Disallow: /directoryname
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Issues with Google Search Console and rekeyed SSL certificate
Hi, Another newbie question please. I've recently changed the name of my business so bought a new domain and rekeyed the SSL certificate to the new domain. Let's say the old domain was called https://123.com and the new one is https://abc.com. I've set up a 301 redirect on 123.com to forward to abc.com and I've added the new domain to Google Search Console and verified it, however can't seem to use the Change of Address tool to move from the old domain to the new domain. I think its because my preferred property (https://123.com) technically no longer exists since I rekeyed the SSL certificate from the old site to the new one so the old site no longer has an SSL certificate. When I go to the old https domain it doesn't load, nor does it seem to forward to the new site. It just times out. Am I correct in assuming that since I rekeyed the SSL certificate, that my original preferred property on Google (https://123.com) no longer exists? And if so, is there a way to use the Change of Address tool or do I simply need to remove the old site from Google and go through a period where my (new) site builds it's ranking from scratch? Thanks in advance folks!
Technical SEO | | Veevlimike0 -
Google Search console says 'sitemap is blocked by robots?
Google Search console is telling me "Sitemap contains URLs which are blocked by robots.txt." I don't understand why my sitemap is being blocked? My robots.txt look like this: User-Agent: *
Technical SEO | | Extima-Christian
Disallow: Sitemap: http://www.website.com/sitemap_index.xml It's a WordPress site, with Yoast SEO installed. Is anyone else having this issue with Google Search console? Does anyone know how I can fix this issue?1 -
Adding /es version to google search console
I have a Wordpress site and we are using WPML for making it bilingual. The domain is: https://www.designerfreelance.net and for Spanish https://www.designerfreelance.net/es Do I have to add to Google search console the /es version? And the no www: https://www.designerfreelance.net https://www.designerfreelance.net/es https://designerfreelance.net https://designerfreelance.net/es and do I have to add the non ssl version? http://www.designerfreelance.net http://www.designerfreelance.net/es http://designerfreelance.net http://designerfreelance.net/es Thanks
Technical SEO | | Trazo0 -
Google place listings and search results- quick question.
Has anybody else noticed that they are ranking better on 'places' yet they have dropped off in the actual search results? We've had no message through webmaster tools. The same seems to have happened to our competitors.
Technical SEO | | onlinechester0 -
Removing a URL from Search Results
I recently renamed a small photography company, and so I transferred the content to the new website, put a 301-redirect on the old website URL, and turned off hosting for that website. But when I search for certain terms that the old URL used to rank highly for (branded terms) the old URL still shows up. The old URL is "www.willmarlowphotography.com" and when you type in "Will Marlow" it often appears in 8th and 9th place on a SERP. So, I have two questions: First, since the URL no longer has a hosting account associated with it, shouldn't it just disappear from SERPs? Second, is there anything else I should have done to make the transition smoother to the new URL? Thanks for any insights you can share.
Technical SEO | | williammarlow0 -
Increase Search Ranking for CEO
Hi guys My company CEO is concerned that when her name is googled pictures of a glamour model appear in the image results area. The glamour model shares a second name with our CEO and this is why the model's images are appearing. I have been asked to rectify this situation. My CEO has a linked in page and twitter account which are underused but no personal page on our company website. I was thinking of buying the url for the CEO's name and optimizing a small site for her name with bio etc and links to twitter, lined in etc. Would this be the best strategy? Thanks Gavin
Technical SEO | | gavinr0 -
Directory URL structure last / in the url
Ok, So my site's urls works like this www.site.com/widgets/ If you go to www.site.com/widgets (without the last / ) you get a 404. My site did no used to require the last / to load the page but it has over the last year and my rankings have dropped on those pages... But Yahoo and BING still indexes all my pages without the last / and it some how still loads the page if you go to it from yahoo or bing, but it looks like this in the address bar once you arrive from bing or yahoo. http://www.site.com/404.asp?404;http://site.com:80/widgets/ How do I fix this? Should'nt all the engines see those pages the same way with the last / included? What is the best structure for SEO?
Technical SEO | | DavidS-2820610