How to Hide Directories in Search?
-
I noticed bad 404 error links in Google Webmaster Tools and they were pointing to directories that do not have an actual page, but hold information.
Ex: there are links pointing to our PDF folder which holds all of our pdf documents. If i type in , example.com/pdf/ it brings up a unformated webpage that displays all of our PDF links.
How do I prevent this from happening. Right now I am blocking these in my robots.txt file, but if i type them in, they still appear.
Or should I not worry about this?
-
Yes, a visit to example.com/dir should now return a 404 error (if you haven't done any redirecting/canonicalizing). This will increase your 404 count in Web Master tools but it's far preferable to the alternative. If you're not redirecting the robots.txt will eventually work and hopefully the links will just fall out of WMT.
-
My hosting company turned off directory browsing and now everything is how it should be. So to my understanding, if the server sees a file that does not have a index file, it should not be view able and should be forbidden. This shoujld not affect us from an SEO standpoint should it? My hosting company said they disabled all directories in our site, however everything still works, except for the forbidden file directories.
-
Basically it shouldn't really have an affect; those unformatted file listings are literally the web server automatically saying 'here's the files that are in this folder', there's no meta tags, description, on page elements, etc.
If you have these pages and they're ranking well, you generally don't want them to be. The automatic file browsing pages don't have your name, your company, etc. in them, and they're generally pretty ugly. They also theoretically could be 'stealing' juice from your 'real' pages, if your internal structure isn't flowing relevance properly.
Basically what I'm saying is that if these pages are having some kind of SEO effect, you probably don't want them to be since they're so basic.
Also I can't overstate the security concerns that directory browsing might be introducing. If someone can directory browse to where your code lives (.php, .aspx.vb, whatever) they may be able to read it. Code sometimes has important things like logins, passwords, merchant account ids, etc. in it that you definitely don't want people reading.
-
Agreed with Valerie that step 1 is to turn off those directory listing pages - that can be a security issue and you don't necessarily want people to see/access the whole list. Also, make doubly sure you don't have any internal links to that directory (Google crawled it somehow).
Generally, Robots.txt should prevent crawling, but it's not foolproof, and it's pretty bad about removing pages once they're indexed. If you can block the page from browsing and return a 404 for the root page, that should be fine. The other option would be to have the page removed in Google Webmaster Tools. You could request removal for the entire folder, but I'm guessing that you may want the actual PDFs indexed.
-
Will turning of directory browsing affect Search for all directories?
-
I really don't want to 301 redirect them as they are just holding files. This is happening with my includes file too. that holds our header, footer, navigation etc. I can check with our hosting company to find out.
-
I'd create an index.html for the directory, and then redirect it somewhere. This way, you're capturing the inbound links and then rescuing some of the inbound juice.
Otherwise, you can also check out this post for more info on other solutions and modifying your htaccess file to prevent the directory view - http://perishablepress.com/better-default-directory-views-with-htaccess/
-
Blocking it in robots.txt will work to hide it from search engines.
If you want to hide it from users or people to who type in the url, you can simply drop a blank "index.html" in the /pdf folder.
-
I would suggest 301'ing them to their /index.htm or /pdf.htm equivalents. If you don't know, a 301 is a signal to a web browser (or search crawler) saying "this page has permanently moved, please go to (otherpage.htm) instead".
Here's a good SEOMoz article explaining it a bit more:
http://www.seomoz.org/learn-seo/redirection
What might be more of a concern, is it sounds like your web server has directory browsing enabled. This could be a security issue (depending on your web server setup). Generally you don't want to expose directories if you don't have to because it gives a potential attacker insight into your system setup. Here's an example how to do it in Apache:
www.camelrichard.org/topics/Apache/Turn_OffDirectoryBrowsing
And IIS:
technet.microsoft.com/en-us/library/cc731109(v=ws.10).aspx
If you like I can confirm if you have open directories if you give me the link, either here or through private message.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Huge Search Traffic Drop After Switching to HTTPS - No Recovery After Couple of Months
Hi In November, we have switched our website (https://www.insidermonkey.com) from HTTP to HTTPS. Initially, we noticed slight search traffic loss but later discovered it might be due to HTTPS switch. A month later we added the https version at search console, and then saw an immediate huge drop (about 25-30%). We discovered the problem might be due to poor redirection and noticed our redirects were 302s instead of 301s. To fix the problem, we implemented the 301 redirects and submitted the sitemap containing links to the old site at the new search console property (https). We've gone through points listed on the page below: https://support.google.com/webmasters/answer/6073543 We fixed the redirects to 301 Double-checked the sitemaps Made sure we had a properly installed SSL certificate (Now, we get A+ from https://www.ssllabs.com/ssltest/analyze.html?d=www.insidermonkey.com) Made sure we have no mixed-content errors (we don't have any issues at search console.) We only avoided implementing HSTS, in case we might want to switch back to HTTP.
Intermediate & Advanced SEO | | etakgoz
We had a small improvement in the following month, but our traffic did not fully recover. We wanted to test for the possibility to switch back HTTP by switching only 2 articles in our CMS to HTTP. Our traffic got worse, not only for those but for the whole site. Then we switched back those 2 articles to HTTPS again and implemented HSTS. It seems our search traffic getting worse day by day with no sign of improving. In the link below you can find the screenshot of our weekly search traffic between 1 October - 1 March. We are down from 500K weekly visitors to mere 167K last week. https://drive.google.com/open?id=1Y1TQbj_YtGG4NhLORbEWbvITUkGKUa0G Any ideas or suggestions? We are willing to get professional help as well. What is the way to find a proper consultant for such problem with relevant experience?0 -
Search engine simulators are not finding text on my website. Do I have a problem with Javascript or AJAX?
My website text is not appearing in search engine simulators. Is there a problem with the javascript? Or perhaps AJAX is affecting it? Is there a tool I can use to examine how my website architecture is affecting how the site is crawled? I am totally lost. Help!
Intermediate & Advanced SEO | | ecigseo0 -
Good Morning America Appearance - Search Rankings Down
We had some products on the Steals and Deals segment of Good Morning America. The same day we received a message from Google in Webmaster Tools (below). The message says that search result clicks have increased significantly. It seems like this was almost a warning that they were not sure this was valid. The promotion included a link from the good morning america site on yahoo to a subdomain on our site. The rankings have fallen a good little bit since and in Webmaster tools, there are no links to our site listed and no internal links and no content keywords for the site. Is this is a temporary freeze on our site until they figure out if this is manipulative? I would have thought a link from Good Morning America would be great for SEO. Search results clicks for http://www.justjen.com/ have increased significantly. This message is not indicative of any problem in your site. It is simply to inform you that the number of clicks that one of your pages receives has increased recently. If you have just added new content, this may indicate that it has become more popular on Google. The number of clicks that your site receives from Google can change from day to day for a variety of factors, including automatic algorithm updates.
Intermediate & Advanced SEO | | gametv0 -
Best Way to Optimize 38 Local Directory Listing In Major Directories
Hi Folks, I am trying to figure out the best way to get our company's 38 U.S. locations in the major local directories. To start, I'd like to get us listed in the major ones: Google, Yahoo, Bing, and Yelp. I do have the resources myself here on staff to do everything manually. So, I don't necessarily need a service like Yext (but would also like any opinions on that offering if anyone can offer it). But, from what I know in the past, every time you try to claim a local listing within each platform, you have to confirm your existence there somehow - whether it be by a mailed postcard or some sort of automated call they give you. Considering that we want to manage all social and local platforms here at corporate, how can we do this? I am not physically at these locations, but I'm sure this is possible to manage everything through one account. The addresses will be local, but the phone numbers on each local profile will route to our customer service here at corporate because the local locations are mostly administrative. In other words, businesses is booked through corporate and carried out at local destinations. Thoughts/Comments?
Intermediate & Advanced SEO | | CSawatzky
I want to do what's best for SEO and also dont' want to harm anything or our link equity. Thanks,
Pedram0 -
Local search vs. Organic Listings
Hi ~ I was interested to see if anyone feels there might be an advantage to keeping a business out of Google's Local Search listing area or at least trying to keep it out of the 7-pack display? It seems to me that sites who are not listed in the 7-pack can often be ranked above the maps/7-pack area in the regular organic listings. Also, is there anyway for a homepage to be listed on the 1st page in both the local search and organic listings? Thanks!
Intermediate & Advanced SEO | | hhdentist0 -
It's a good idea to have a directory on your website?
Currently I have a directory on a sub domain but Google apparently sees it as part of my main domain so all outgoing links may be affecting my rankings?
Intermediate & Advanced SEO | | Valarlf0 -
Increasing Search Queries
Recently I had a drop in the over all number of search queries my website was ranking for (about 50%) on October 5th. I did not lose rankings for my target keywords. How can I regain these lost opportunities?
Intermediate & Advanced SEO | | raph39880 -
In mobile searches, does Google recognize HTML5 sites as mobile sites?
Does Google recognize HTML5 sites using responsive design as mobile sites? I know that for mobile searches, Google promotes results on mobile sites. I'm trying to determine if my site, created in HTML5 with responsive design falls into that category. Any insights on the topic would be very helpful.
Intermediate & Advanced SEO | | BostonWright0