How to prevent directory from being accessed by search engines?
-
Pretty much as the question says, is there any way to stop search engines from crawling a directory? I am working on a Wordpress installation for my site but don't want it to be listed in search engines until it's ready to be shown to the world. I know the simplest way is to password-protect the directory but I had some issues when I tried to implement that so I'd like to see if there's a way to do it without passwords. Thanks in advance.
-
But don't forget to remove that Disallow out of Robots.txt when you go live - if you want those pages to be indexed (and also the Meta-robots noindex nofollow).
Otherwise you might be pulling your hair out trying to figure out why none of your pages are getting indexed in the SERPs.
-
You're absolutely right! I left that part out. Thanks
-
The robots.txt file does not guarantee that your pages will not show up in search results! Your best bet after password protection is adding a NoIndex meta tag to you page headers.
Google have openly said that they obey this tag (Matt Cutts).
-
Xee,
It always help, and it is very easy to implement. This function to show the path to the sitemap ir very good.
-
It's not required to have the ending slash. At least, it works for us without it.
-
As it is, my site is just phpBB3 forums (www.bearsfansonline.com); would a sitemap really help that much?
-
If you don't have an robot.txt file, you need to include some important stuff first.
First, do you have a sitemap.xlm for your website? If not, its very important and you should creat it at: http://www.xml-sitemaps.com/
Create a robot.txt file and include the follow:
User-agent: * allow: / disallow: /directoryname
Sitemap: http://www.yousite.com/sitemap.xmlWith this you will inform all robots where is your sitemap. You should read more about robots.txt in this great post: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts
-
shouldn't you put a slash at the end of the directory in the robots file?
you can create the robots file through the Google Webmaster Tools
-
I don't have a robots.txt file in my root. Do I just create a text file, put the above lines into it, and upload it to my root after changing the name?
-
I'm assuming you want all search engines blocked from this directory. If so, edit your robots.txt file to state the following. This will block all bots from accessing a folder/directory on your site
User-agent: *
Disallow: /directoryname
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Australian search - ZERO visibility and stumped
Fair warning, this is going to be long, but necessary to explain the situation and what has been done. I will take ANY suggestions, even if I have tried them already. We have a sister site in Australia, targeting Australian traffic. I have inherited what seems to be an incredible rat's nest. I've fixed over two dozen issues, but still haven't seemed to address the root cause. NOTE: Core landing pages have weak keyword targeting. I don't expect much here until I fix this. The main issues I'm trying to resolve first are with the unusual US-based targeting, and the inability of the homepage to rank for anything. The site is www[dot]castleford[dot]com[dot]au. Here's the rundown on what's going on: Problems: The site ranks for four times as many keywords in the US as it does in Australia. The site ranks for a grand total of 5 keywords on the first page for AU keywords. The homepage, while technically optimized on-page for "content marketing agency", and with content through MarketMuse, has historically ranked between 60-100, despite having a fairly strong DA with fairly weak competitors, based on AHREFs keyword difficulty, and Moz keyword difficulty. Oddly, the ranking has gone up to 5-7 for three day spurts over the past year. Infrequent indexing of homepage (used to be every 2-3 weeks, I've gotten that down to 1 week). Sequence of events: November 2017 - they made some changes to their URLs - some on the blog and some on the top nav LPs. Redirects seem okay. November 2017 - Substantial number of lost referring domains, not many seem to be quality. January 2018 - total number of AU ranking keywords more than halved. May/June 2018 - added a follow inbound link sitewide to an external site that they created. 20k inbound links with same anchor text to homepage. Site has a total of 24k inbound links. July-Sep 2018 - total number of US ranking keywords halved November 10 - I walked into this mess. What's been done: Reduced site load speed by over 150% (it was around 20 seconds). Create sitemap (100 entry batching) and submit to GSC. Improved MarketMuse score for the homepage. Changed language from "en-US" to "en-AU" Fetch and render - content is all crawlable and indexed properly. Changed site architecture for top nav core landing pages to establish clear hierarchy. All version of GSC created, non-www and www http, and non www https and www https Site crawl - normal amount of 404s, nothing stands out as substantial. http to https redirect okay. Robots.txt updated and okay. Checked GSC international targeting, confirmed AU. No manual links penalty I'm clearly stumped and could use some insights. Thanks to everyone in advance, if you can find time.
Technical SEO | | Brafton-Marketing0 -
Wordpress vs. home grown directory
I just moved my blog from a wordpress hosted solution to my owner server and am really hopeful that all the SEO ready wordpress pages plus I downloaded Yoast SEO will move my site. I started with 9000 pages being moz indexed with tons of errors eyerywhere, I have almost fixed everything getting ready to do a google index. One question. I have a directory of businesses and events for my local mountain community that is home grown with about 200 pages. see here: http://www.destinationbigbear.com/directory/bigbeardirectory.aspx which has a PA of 21 which is the same as my DA 21 (I know terrible)... Should I migrate these 200 pages with images to wordpress to take advantage of all the nice SEO possibilities? I have staff and it would probably take about $600 bucks to do it. I would only have to put about ten 301's such as http://www.destinationbigbear.com/directory/contentcat.aspx?ParentID=7 would be http://www.destinationbigbear.com/big-bear-restaurants/ Thank you again to all, I am hopeful I can answer some questions in the future for people... I am learning alot! Nick
Technical SEO | | nickcargill0 -
No-follow for article directory?
My clients pull from a central article directory on our server (medical directory), as the information is about standard medical issues. This said, the MOZ analytics is showing these articles for each client as indexed and duplicate in content, descriptions, titles, etc. Would it be better to use a no-follow for these articles to avoid looking like duplicate content, or should I consider overhauling the resource section into static pages and making each article unique to each client-considering the latest updates in Google? Any help/insight would be greatly appreciated!!!!! Thanks
Technical SEO | | lfrazer0 -
Google webmaster tools says access denied error 403
Hi, this keeps on happening, just check early today and it tells me i have access denied and 403 errors I have this from time to time in my google webmaster tools and i have checked the pages and they work properly, so i am puzzled why this has happened. I have contacted my hosting company who have said there is not a problem but there must be a problem somewhere which could affect my site rankings. can anyone let me know what this could be please. i work in joomla | parenting-magazine | 403 | 8/10/13 |
Technical SEO | | ClaireH-184886
| | 2 | personal-finance-money-advice | 403 | 8/10/13 |
| | 3 | 201308081607/emmerdale/emmerdale-chas-confronts-cameron-over-affair-with-debbie | 403 | 8/10/13 |
| | 4 | 201308081606/emmerdale/emmerdale-declan-gets-a-visit-from-the-police | 403 | 8/10/13 |
| | 5 | 201308081608/emmerdale/emmerdale-cameron-debbie-affair-is-out-in-the-open | 403 | 8/10/13 |
| | 6 | 201308081614/uk-holiday-news/visitscotland-launch-campaign-to-boost-tourism | 403 | 8/10/13 |
| | 7 | dog-advice/training-your-puppy-a-beginners-guide | 403 | 8/10/13 |
| | 8 | gadgets/hp-envy-13-laptop-review | 403 | 8/10/13 |
| | 9 | gadget-talk/everyday-smartphone-gadgets-which-could-revolutionise-your-life | 403 | 8/10/13 |
| | 10 | news-gadgets/the-htc-one-mobile-phone-review | 403 | 8/10/13 |
| | 11 | gadget-talk/five-iphone-apps-for-home-improvement | 403 | 8/10/13 |
| | 12 | gadget-talk/are-android-apps-useful-for-business-success | 403 | 8/10/13 |
| | 13 | gadget-talk/television-gadgets-the-future-of-television-is-coming | 403 | 8/10/13 | | | |0 -
See Different Landing page for my main keyword in google search result
I have a website like http://www.bannerbuzz.com, i am promoting home page with vinyl banners keyword, but currently i can see my website's review page for vinyl banners result in google, i want to display my home page instead of review page for my keyword result in google, its frequently change, some time i can see home page for it and some time it shows review page as i attached image. i want to show my home page, so can you please help me to solve it, how can i stable my home page with main keywords. OtOXxiE.png
Technical SEO | | CommercePundit0 -
How to add business address in local directories for consistent NAP
Hi Mozers I keep puzzling over this one! I work from home and really don't want to plaster my address all over the web. The GP page now allows for me to hide my exact location, which is great. However, as far as I can see this is not the case with all the potential local directories and listings. I have been trying in to get around this by not adding my house number and last digit and 2 characters of my post code. So far this has been allowed by the local listings I have signed up with. When I tried doing as recommended by the excellent Miriam and checking my business name with 'Getlisted' I found that I could only see these local listings if I added the doctored address, i.e. no house number or full postcode. My question, finally is, if I continue in this fashion for businesses based at home addresses am I going to confuse the search engines. I want to provide a consistent NAP but GPP insists that I add a full postcode. The only way I could possible see around this is to add: street name city full postcode and omit the house name/number. Will this be a reasonable work around to maintain client confidentiality and satisfy the NAP requirement of Local search?
Technical SEO | | catherine-2793880 -
Different pages first results on same keyword search
Hi, Sometimes Google does not show the page you intended for a certain keyword. Logically you would say that the intended page is not relevant/strong enough. But in my case several pages ranked fine for a long period of time and all of a sudden another less important page gets the highest result on a keyword search. (We are in the camping business) For instance: One of our campsites called Tenuta primero used to rank position 9 in google with page below for a long time (search: 'camping tenuta primero') This was the page we intended to rank with. http://www.suncamp.nl/nl/nl/campings/italie/friuli-venezia-giulia/camping-tenuta-primero/uc19-l1-n797-c13-r115-cp104959/ Now all of a sudden the position for search 'camping tenuta primero' is position 33 with review page below. http://www.suncamp.nl/nl/nl/campings/italie/friuli-venezia-giulia/camping-tenuta-primero/beoordelingen/uc19-l1-n797-c13-r115-cp104959-t22598/ What could have caused this? Pages are in Dutch but main keywords are camping are tenuta primero. Thank you very much in advance! Kind regards, Dennis Overbeek Dennis@acsi.eu | www.suncamp.nl |
Technical SEO | | SEO_ACSI0 -
Mobile site ranking in desktop searches
A robots.txt file is properly implemented on the mobile site to block Googlebot from crawling, yet, when running a site: query for the site in google, it still returns over 104,000 pages from the mobile site in the index. Why could this be happening?
Technical SEO | | craigsmith3330