Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hidden Text w/ Java Script _ Is it Bad?
Just came across an article that stated that Google is looking negatively at sites that attempt to hide text or use javascripts to expand text on websites. We are about to launch our new website and believe we are using this technique but im not certain if what we are doing will hurt us. Our website tends to be a little heavy on the text so used a "read more" scrpit that will expand when clicked on. Three sections that use this on the new website Take a look and let me know your thoughts http://joomplateshop.com/demos/catdi.com/
Web Design | | ChopperCharlie0 -
Is it still necessary to have a "home" page button/link in the top nav?
Or is it not necessary to have a "home" tab/link because everybody by this time knows you can get to the home page by clicking on the logo?
Web Design | | FindLaw0 -
Does having a Blog link in the top level navigation provide any better SEO value, or would having it in a footer or top navigation work just as good?
Trying to decide on whether placing a link to the blog in our top level navigation would have a better SEO value than just placing it in top or footer navigation. I have an ecommerce site.
Web Design | | RPD0 -
Redirects (301/302) versus errors (404)
I am not able to convincingly decide between using redirects versus using 404 errors. People are giving varied opinions. Here are my cases 1. Coding errors - we put out a bad link a. Some people are saying redirect to home page; the user at least has something to do PLUS more importantly it does NOT hurt your SEO ranking. b. Counter - the page ain't there. Return 404 2. Product removed - link1 to product 1 was out there. We removed product1; so link1 is also gone. It is either lying in people's bookmarks, OR because of coding errors we left it hanging out at some places on our site.
Web Design | | proptiger0 -
Hom much does getting a mobile website improve the "mobile ranking"?
There's been speculation about Google totally not ranking sites that are not mobile, but as far as I can see many sites rank THE SAME on mobile devices as on regular stationary PCs/lap-tops. I had hoped for a radically improved ranking once a mobile site was built. Or is it just that it takes a TON of time for it to go through and then it gets better or is it like with regular sites that a new mobile site on a sub-domain is parked in some form of mobile-sand-box? Also, does putting a mobile site on a ranking domain, as subdomain help, e.g. m.testingsite.com instead of testingsite.mobi ? Thanks.
Web Design | | yvonneq0 -
Searching for BEST e-Commerce Multilanguage Platform, Need Advice
I have 2 online store in Canada. Both are selling One bilingual (English&french) Filtration Montreal and a unilinguale store Furnace Filters Canada Both are offering the same products at the same price for Canadian. We work hard to rank on Google.ca because we only sell and ship to Canada. The platform of Filtration Montreal is very basic and limited. For example, the url structure make it very hard to rank on Google.ca This platform is very not SEO friendly with url like: http://www.filtrationmontreal.com/en/product/honeywell-genuine-filter-95/pack-of-5-genuine-honeywell-furnace-filters-20x20x5-601.html The only good thing about this platform, is the multilingual option. The customer can shop in french or English. I would like to move that store to a new platform where I can create a multilingual online store. Do you have sugestions? Furnace Filters Canada is on BigCommerce. I find it SEO friendly. Using SEOmoz tools and new to SEO, high competitive keywords like: furnace filters, furnace filter are ranking on 3rd rank in Google.ca fist page! This site is getting more & more visitors every months. The only frustrating thing is the English only version of the stores to customers. QUESTIONS: What will be the SEO impact if I'm moving Furnace Filters Canada to a new platform? Do you have suggestions in finding the perfect multilanguage e-Commerce platform? Andrew Bleakley suggest Ashop. Anybody using Ashop? How about a eCommerce platform that can manage my 2 stores at the same time. REMEMBER, we sell and ship to Canada only. Thank you for your help and support. BigBlaze
Web Design | | BigBlaze2050 -
Optimzing a new ecommerce site, Need help with URL
Hi We are putting up a new ecommerce website and for product description, our tech team indicates that they must have the skun numbers in the URL. Which one of the following URL structure do you find the most SEO freindly? 1. http://www.Site.com/SKUNumber/ProductDescription/ or 2. http://www.Site.com/ProductDescription/SKUNumber/ My personal opinion is that most relevant content should be on load page so I like option 1. Thanks
Web Design | | CookingCom0 -
Can "poor" subdomains drop PR of the root domain?
The page rank of my company's website has dropped from a 6 to a 4 over the past year or so. In that time, we implemented subdomains for development sites to show clients progress on their websites. I noticed that our "dev" sites are being indexed while in development and my question is, will Google drop pagerank of our root domain purely off of these "dev" subdomains? Example - our site is www.oursite.com Dev site - development1.oursite.com I just began investigating the drop and this came to my mind yesterday but am not too sure what type of impact these non-credible subdomains will have on our root domain. Any thoughts?
Web Design | | ckilgore0