How to Hide Directories in Search?
-
I noticed bad 404 error links in Google Webmaster Tools and they were pointing to directories that do not have an actual page, but hold information.
Ex: there are links pointing to our PDF folder which holds all of our pdf documents. If i type in , example.com/pdf/ it brings up a unformated webpage that displays all of our PDF links.
How do I prevent this from happening. Right now I am blocking these in my robots.txt file, but if i type them in, they still appear.
Or should I not worry about this?
-
Yes, a visit to example.com/dir should now return a 404 error (if you haven't done any redirecting/canonicalizing). This will increase your 404 count in Web Master tools but it's far preferable to the alternative. If you're not redirecting the robots.txt will eventually work and hopefully the links will just fall out of WMT.
-
My hosting company turned off directory browsing and now everything is how it should be. So to my understanding, if the server sees a file that does not have a index file, it should not be view able and should be forbidden. This shoujld not affect us from an SEO standpoint should it? My hosting company said they disabled all directories in our site, however everything still works, except for the forbidden file directories.
-
Basically it shouldn't really have an affect; those unformatted file listings are literally the web server automatically saying 'here's the files that are in this folder', there's no meta tags, description, on page elements, etc.
If you have these pages and they're ranking well, you generally don't want them to be. The automatic file browsing pages don't have your name, your company, etc. in them, and they're generally pretty ugly. They also theoretically could be 'stealing' juice from your 'real' pages, if your internal structure isn't flowing relevance properly.
Basically what I'm saying is that if these pages are having some kind of SEO effect, you probably don't want them to be since they're so basic.
Also I can't overstate the security concerns that directory browsing might be introducing. If someone can directory browse to where your code lives (.php, .aspx.vb, whatever) they may be able to read it. Code sometimes has important things like logins, passwords, merchant account ids, etc. in it that you definitely don't want people reading.
-
Agreed with Valerie that step 1 is to turn off those directory listing pages - that can be a security issue and you don't necessarily want people to see/access the whole list. Also, make doubly sure you don't have any internal links to that directory (Google crawled it somehow).
Generally, Robots.txt should prevent crawling, but it's not foolproof, and it's pretty bad about removing pages once they're indexed. If you can block the page from browsing and return a 404 for the root page, that should be fine. The other option would be to have the page removed in Google Webmaster Tools. You could request removal for the entire folder, but I'm guessing that you may want the actual PDFs indexed.
-
Will turning of directory browsing affect Search for all directories?
-
I really don't want to 301 redirect them as they are just holding files. This is happening with my includes file too. that holds our header, footer, navigation etc. I can check with our hosting company to find out.
-
I'd create an index.html for the directory, and then redirect it somewhere. This way, you're capturing the inbound links and then rescuing some of the inbound juice.
Otherwise, you can also check out this post for more info on other solutions and modifying your htaccess file to prevent the directory view - http://perishablepress.com/better-default-directory-views-with-htaccess/
-
Blocking it in robots.txt will work to hide it from search engines.
If you want to hide it from users or people to who type in the url, you can simply drop a blank "index.html" in the /pdf folder.
-
I would suggest 301'ing them to their /index.htm or /pdf.htm equivalents. If you don't know, a 301 is a signal to a web browser (or search crawler) saying "this page has permanently moved, please go to (otherpage.htm) instead".
Here's a good SEOMoz article explaining it a bit more:
http://www.seomoz.org/learn-seo/redirection
What might be more of a concern, is it sounds like your web server has directory browsing enabled. This could be a security issue (depending on your web server setup). Generally you don't want to expose directories if you don't have to because it gives a potential attacker insight into your system setup. Here's an example how to do it in Apache:
www.camelrichard.org/topics/Apache/Turn_OffDirectoryBrowsing
And IIS:
technet.microsoft.com/en-us/library/cc731109(v=ws.10).aspx
If you like I can confirm if you have open directories if you give me the link, either here or through private message.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Large Competitor closed, how to capitalize in search. Any ideas?
Hey Mozzers, One of our biggest competitors closed down on January 1st, 2020 in several US cities. They did stay open in some areas just FYI. The competitor's website is www.execucar.com. This is a very large company that has a presence in almost all US major airports. It's a private car service just like Uber but for wealthy individuals. For example. when you search " lax car service" they are #3 on Google or "car service to lax" they're #2 still. What can we do to get more of their traffic and actual business? Has anyone done something like this before or knows quick and easy tactics to get their clients? We have a local landing page: https://dcacar.com/lax-car-service that ranks 9 through 11 for those same keywords. Thanks for your thoughts and time. Davit
Intermediate & Advanced SEO | | Davit19850 -
How to Improve Search Ranking from 10 to Top 5
Hi, I was just optimizing a page and have improved it from Page 2 -> Page 1 ; Position 15->10 for now by basic onpage seo edits. I want to understand how I would take it to next level by getting it to Top 5 or Top 3 results. My keyword and page are as follows (all checked in Google India (.in) ) Page - https://nirogam.com/ayurvedic-treatment-home-remedies-chikungunya/
Intermediate & Advanced SEO | | pks333
Keyword - ayurvedic treatment for chikungunya What steps can I take to go from 10 - Top 3 or Top 5 results here? I was checking the Rank tracker & moz Grader for this - got all ticks except adding keyword in URL. Would this be recommended of changing the URL after it's ranking so well to just add this keyword in the same?0 -
Crawled page count in Search console
Hi Guys, I'm working on a project (premium-hookahs.nl) where I stumble upon a situation I can’t address. Attached is a screenshot of the crawled pages in Search Console. History: Doing to technical difficulties this webshop didn’t always no index filterpages resulting in thousands of duplicated pages. In reality this webshops has less than 1000 individual pages. At this point we took the following steps to result this: Noindex filterpages. Exclude those filterspages in Search Console and robots.txt. Canonical the filterpages to the relevant categoriepages. This however didn’t result in Google crawling less pages. Although the implementation wasn’t always sound (technical problems during updates) I’m sure this setup has been the same for the last two weeks. Personally I expected a drop of crawled pages but they are still sky high. Can’t imagine Google visits this site 40 times a day. To complicate the situation: We’re running an experiment to gain positions on around 250 long term searches. A few filters will be indexed (size, color, number of hoses and flavors) and three of them can be combined. This results in around 250 extra pages. Meta titles, descriptions, h1 and texts are unique as well. Questions: - Excluding in robots.txt should result in Google not crawling those pages right? - Is this number of crawled pages normal for a website with around 1000 unique pages? - What am I missing? BxlESTT
Intermediate & Advanced SEO | | Bob_van_Biezen0 -
Consolidating numerous landing pages using similar search terms
Hi, The site I am working on currently uses numerous pages for search terms with similar keywords. vehicle wrapping / vehicle wraps / car wrapping / car wraps / van wrapping / van wraps etc Now obviously i want to bring these into one to help create one high authority page covering all terms. At present the "car wraps" page is ranking for quite a few of these terms. Am i best to stick with this or chose the highest search term being car wrapping, and pass the dribbles of juice from the rest and "car wraps" onto this? This is aimed at a local demographic so the local terms will be thrown in too unless you think the places pages will work in favour? Many thanks,
Intermediate & Advanced SEO | | Lee4dcm0 -
Monthly Searches from Google Keyword Planner
I've used Google Adwords, Google Analytics and competitors keywords to compile a master list. I'm now looking to evaluate metrics on the keywords / phrases / long tail phrases. My question is this ... Based on Googles use of Geo Targeting, would I be better to evaluate metrics (Avg. Monthly Searches, Competition, Avg CPC) based on United Kingdom or my local city (I only operate in my local city). I am looking to use the results to redesign my website. I will use the favorable keywords / phrases / long tail keywords to implement a new menu, new content page creation, articles, etc. Thanks Mark
Intermediate & Advanced SEO | | Mark_Ch0 -
Language Attribute - does changing it make a difference to SEO and Search?
I am an SEO newbie, but learning fast. 🙂 I am based in London, UK and have a website: www.twofourseven.co.uk. I noticed that the language attribute was set to 'en-US'. I work in London as well in international locations in the Middle East and Asia. Thinking of this I wanted to ask the experts if given that I am based in the UK, would changing the language attribute make a difference to search results? If so, would 'en' be better than 'en-GB', which might be too specific? Thanks in advance!
Intermediate & Advanced SEO | | twofourseven0 -
Transactional vs Informative Search
I have a page that id ranking quiet good (Page1) for the plural of a Keyword but it is just ranking on Page 3 for the Singular Keyword. For more then one Year I am working on Onpage and Offpage optimization to improve ranking for the singular term, without success. Google is treating the two terms almost the same, when you search for term one also term 2 is marked in bold and the results are very similar. The big difference between both terms is in my opinion that one is more for informational search the other one is more for transactional search. Now i would be curious to know which factors could Google use to understand weather a search and a website is more transactional or informative? Apart of mentioning: Buy now, Shop, Buy now, Shop, Special offer etc. Any Ideas?
Intermediate & Advanced SEO | | SimCaffe0 -
Sub domain versus separate domains, which is better for Search engine purposes?
We are pitching to a hotel client to build two new websites, a summer website and a winter website, two completely different looking websites. The client wants to automatically switch their domain name to point to one or the other, depending on the time of year. The customer does not want to use a landing page where you would choose which site to visit; they want the domain name to go directly to the relevant website. Our options: Set up two new domain names and optimise each website based on the holiday season and facilities offered at that time of year. Then change the exisiting domain name to point at the website that is in season. Or Use the existing domain name and setup two sub domains, switching the home page as necessary. We have been chewing this one over for a couple of days, the concern that we have with both options is loss of search visibility. The current website performs well in search engines, it has a home page rank of 4 and sub-pages ranking 2 and 3’s, when we point the domain at the summer site (the client only has a winter website at present) then we will lose all of the search engine benefits already gained. The new summer content will be significantly different to the winter content. We then work hard for six months optimising the summer site and switch back to the Winter site, the content will be wrong. Maybe because it's Friday afternoon we cannot see the light for the smoke of the cars leaving the car park for the weekend, or maybe there is no right or wrong approach. Is there another option? Are we not seeing the wood for the trees? Your comments highly welcome. Martin
Intermediate & Advanced SEO | | Bill-Duff0