How to Disallow Specific Folders and Sub Folders for Crawling?
-
Today, I have checked indexing for my website in Google. I found very interesting result over there. You can check that result by following result of Google.
I aware about use of robots.txt file and can disallow images folder to solve this issue.
But, It may block my images to get appear in Google image search.
So, How can I fix this issue?
-
You can, but then the content will be removed from Google's index for 90 days. I am not sure what effect this would have on pages with the images. It shouldn't have any effect, but I would hate for you to have rankings in any way affected for 90 days.
I have no experience in having images indexed in this manner. Perhaps someone else has more knowledge to share on this topic.
-
Can I use Remove URL facility from Google webmaster tools?
-
I checked your URL: http://www.lampslightingandmore.com/images/. The folder is now properly restricted and the images can no longer be seen using this method. Going forward, Google will not be able to index new images in the same manner your other images were indexed.
With respect to the images which have been indexed, I am not certain how Google will respond. The image links are still valid so they may keep them. On the other hand, the links are gone so they may remove them. If it were my site, I would wait 30 days to see if Google removed the results.
Another way you can resolve the issue is to change the file path to your images from /images to /image. This will immediately break all the links. You would need to ensure all the links on your site are updated properly. It still may take Google a month to de-index those results but it would certainly happen in that case.
-
I have added Options -Indexes for images folder in htaccess file.
But, I still able to find out images folder in Google indexing.
Can I check? Is it working properly or not? I don't want to index or display images folder in web search any more.
-
I am going to add following code to my htaccess page.
Options -Indexes
Will it work for me or not?
-
If you have a development team, they should instantly understand the problem.
A simple e-mail to any developer
E-mail title: Please fix
http://www.lampslightingandmore.com/images/
That's it. No other text should be needed. A developer should be able to look at the page and understand the index was left open and how to fix it. If you wish to be nicer then a simple "my index is open for the world to see, please don't allow public access to my server folders" should suffice.
-
Yes, I have similar problem with my code structure. Yesterday, I have set Relative path for all URLs. But, I am not sure about replacing of image name in code after make change in folder.
So, I don't want to go with that manner. I also discussed with my development team and recommend to go with htaccess method.
But, give me caution to follow specific method otherwise it may create big issue for crawling or indexing. Right??
-
The link you shared is perfect. Near the top there is a link for OPTIONS. Click on it and you will be on this page: http://httpd.apache.org/docs/1.3/mod/core.html#options
I want to very clearly state you should not make changes to your .htaccess file unless you are comfortable working with code. The slightest mistake and your entire site becomes unavailable. You can also damage the security of your site.
With that said, if you decide to proceed anyway you can add the text I shared to the top of your .htaccess file. You definitely should BACK UP the file before making any changes.
The suggestion vishalkialani made was to rename your /images folder to something else, perhaps /image. The problem is that if your site was not dynamically coded, you would break your image links.
-
In addition to what Ryan mentioned I would rename that folder on your server. That will make google's index outdated and you won't get any visitors on the server
-
I can't getting you.
-
also you can rename it so when google 's index shows up the results you won't get any hits.
if thats what you want.
-
Yes, I checked article to know more about it.
http://httpd.apache.org/docs/1.3/howto/htaccess.html
But, I am not able to find my solution. Can you suggest me specific article which suppose to help me more in same direction?
-
Hello.
You have left your site open in a manner which is not recommended. Please take a look at the following URL: http://www.lampslightingandmore.com/images/. On a properly secured server, you should receive a 404 Page Not Found or Access Denied type of error. Since the folder is left open, a Google crawler found it and you are seeing the results.
The means to secure your site varies based on your software configuration. If you are on an Apache web server (the most common setup) then these settings are controlled by your htaccess file. I am not an htaccess expert but I believe adding the following code to your .htaccess file at the top will fix the issue:
Options -Indexes
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO strategies specifically for saas companies
For location specific businesses there are many tactics to locally optimize your business letting them rank higher than large companies. Are there any tactics/strategies specifically for saas companies?
Intermediate & Advanced SEO | | alexadedmon0 -
Sub-directories or Nah
Hey MOz Squad, So I have a locksmith company with 7 locations all up and down the west coast. We set up each location in a sub-directory (mainly because they used to have other websites for these locations and we wanted to try and keep the juice so we 301 redirected back to these subdiretories) But it's making our efforts so tough because every time I want to change something I have to do it seven times! DO the sub directories really matter that munch for rank? One of my partners says google treats it like it own website and trying to rank just a page on a website for a certain city is harder. What do you guys think Keep the subs or ditch em?
Intermediate & Advanced SEO | | Meier0 -
How can I make sure Google is crawling a link from an iframe (video)?
Do they crawl backlinks from an iframe example from a Youtube video embedded in a blog post? TIA!
Intermediate & Advanced SEO | | zpm20140 -
Should sub domains to organise content and directories?
I'm working on a site that has directories for service providers and content about those services. My idea is to organise the services into groups, e.g. Web, Graphic, Software Development since they are different topics. Each sub domain (hub) has it's own sales pages, directory of services providers and blog content. E.g. the web hub has web.servicecrowd.com.au (hub home) web.servicecrowd.com.au/blog (hub blog) http://web.servicecrowd.com.au/dir/p (hub directory) Is this overkill or will it help in the long run when there are hundreds of services like dog grooming and DJing? Seems better to have separate sub domains and unique blogs for groups of services and content topics.
Intermediate & Advanced SEO | | ServiceCrowd_AU0 -
Is having all your media hosted on a sub-domain bad?
I just realized yesterday while doing some audit work on our site (which is still relatively new) that all of our audio assets are stored on a separate sub-domain. We are an eCommerce site that sells audio books, and every product page has a sample audio file to listen to. But all those files are stored on a sub-domain of the main site. "cdn-media.oursite.com". First, I understand that media(our audio files) has some inherent SEO value if hosted correctly. Is that true? And if so, how important would you think it is? Secondly, assuming that it does have value, are we losing that value by having them hosted on a sub-domain? I have read things that say sub-domains are bad, and I have read things that say that Google at least has been treating sub-domains as sub-folders, but I can't find anything definitive one way or the other. On another note, another thing I saw is that people are linking to the sound files directly in various places, and those links are going to the sub-domain, not the main domain. There aren't even pages on the sub-domain, just the files, so those links deliver a "visitor" to a page that is completely blank except for a tiny little audio player. Not sure what to do about that, but that can't be good one way or the other right? How big of a problem is this really? Is it worth me going to our IT dept. and trying to change it? It sounds like it would be a pretty big deal to change, so I'll need a few voices to back me up if that's the case.
Intermediate & Advanced SEO | | DownPour0 -
What impact will new folder structure have?
I'm evaluating a request to restructure our existing folder structure for Ties.com and I would like some input. I typically recommend proper folder structure for ecommerce sites either when doing a new build or when doing an information architecture overhaul because of duplicate content or canonicalization issues. Ties.com doesn't have duplicate content or canonicalization issues so I'm really trying to get feedback about the SEO impact of a folder structure change versus the amount of time required on our side to do the work. Please share your experiences. Current URL structure example: http://www.ties.com/v/a/the-american-necktie-co-navy-and-gold-stripe-navy-blue-skinny-tie Potential New Structure: http://www.ties.com**/skinny-ties**/the-american-necktie-co-navy-and-gold-stripe-navy-blue-skinny-tie OR http://www.ties.com/skinny-ties/brand/product-title I'm well aware that as a BEST PRACTICE we should be using the category/subcategory folder structure, but since we aren't really suffering from any technical SEO issues then I don't know if this will be worth our time to remap all of these URL's via 301 and canonicalize everything. I anticipate it will be a lot of work and if the Return on Effort is low I can't prioritize this project. I need 2nd and 3rd opinions from experienced ecommerce retail SEO's. Thank you
Intermediate & Advanced SEO | | Ties.com0 -
Correlation Between Domain Authority and Crawl Penetration?
A. Is there a correlation between domain authority and crawl penetration? B. Is there a correlation between domain authority and juice distribution?
Intermediate & Advanced SEO | | AWCthreads0