Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Having a Subfolder/Subdirectory With a Different Design Than the Root Domain
Hi Everyone, I was wondering what Google thinks about having a subfolder/subdirectory with a different design than the root domain. So let's say we have MacroCorp Inc. which has been around for decades. MacroCorp has tens of thousands of backlinks and a couple thousand referring domains from quality sites in its industry and news sites. MacroCorp Inc. spins off one of its products into a new company called MicroCorp Inc., which makes CoolProduct. The new website for this company is CoolProduct.MacroCorp.com (a subdomain) which has very few backlinks and referring domains. To help MicroCorp rank better, both companies agree to place the MicroCorp content at MacroCorp.com/CoolProduct/. The root domain (MacroCorp.com) links to the subfolder from its navigation and MicroCorp does the same, but the MacroCorp.com/CoolProduct/ subfolder has an entirely different design than the root domain. Will MacroCorp.com/CoolProduct/ be crawled, indexed, and rank better as both companies think it would? Or would Google still treat the subfolder like a subdomain or even a separate root domain in this case? Are there any studies, documentation, or links to good or bad examples of this practice? When LinkedIn purchased Lynda.com, for instance, what if they kept the https://www.lynda.com/ design as is and placed it at https://www.linkedin.com/learning/. Would the pre-purchase (yellow/black design) https://www.linkedin.com/learning/ rank any worse than it does now with the root domain (LinkedIn) aligned design? Thanks! Andy
Web Design | | AndyRCWRCM1 -
Is this spammy/panda problem?
We have a property site and have many area pages with listings of properties. For example, one page may have up to 30 properties listed, with 100 words of description on that listing page for that property and then you click 'more info' to get to the page of that property with maybe 200 words in total. I want to add bullet points for each property on the area page but Im worried that it may be seen by google as spammy even though its usefull to the client. For example, if I had 30 properties on a page, and 28 of them said next to each picture.. Property Type: Shared, Open Plan, Single Bed Would that be a problem for google?
Web Design | | nick-name1230 -
What are the downsides and/or challenges to putting page paths (www.example.com/pagepath) on a different server?
Hi, Our company is organized into three different segments and our development team recently needed to switch a portion of the business to subdomain because they wanted to move to a different server platform. We are now seeing the impact of moving this segment of the business to a subdomain on the main domain. SEO is hurting and our MOZ score has dropped significantly. One fix they are debating is moving everything back to one domain, but place segments of the business on different page paths and hosting specific paths on different servers. I.e. the main domain could be www.example.com hosted in one location and then www.example.com/segment1 would be hosted on a different server. They are hoping to accomplish this using some sort of proxy/caching redirection solution. The goal of this change would be to recapture our domain strength. Is this something that is a good option or no? If not, what are the challenges and issues you see arising from doing something like that as I don't know of any other site set up like this. Thanks in advance.
Web Design | | bradgreene0 -
Does using role="heading" instead of H1 in HTML code affects SEO?
Does using role="heading" instead of affect SEO? http://www.w3.org/WAI/GL/wiki/Headings_using_role%3Dheading
Web Design | | LNEseo0 -
Blog vs. News/Editorial Layout?
We're in the coupon blogging space & saw that one of the larger coupon sites move away from the more traditional blog layout: http://thekrazycouponlady.com they now have more of an editorial type layout. Here is another site which is more similar to our layout: http://hip2save.com. So here are my questions: Which layout type do you feel better serves their visitors & why? How does the affect the SEO of the site? How does it affect the advertising revenues? Which layout do you prefer? Is there strategy in this move for the coupon blog, or is this just a preference on how they now display their content? We're making some updates to our design soon & I wanted to get some feedback on the overall direction we take.
Web Design | | seointern0 -
Is it common to have some of error/warning(currency duplicate,redirect, etc...) in most website that rank well?
Hi could any body could give me some idea on 'on page optimisation' Currently in my campaign I have around 3000+ errors, 14,000+ warning, 7000+ notices for the following reasons: Overly-Dynamic URL
Web Design | | LauraHT
Temporary Redirect
Title Element Too Long (> 70 Characters)
Duplicate Page Title
etc... First of all I know these have negative effect on SEO. Now to fix towards those issues it involve lots of works and times. At the same time most of our important keywords/url rank position have not changed over the last 12 months. Does that mean the above has only limited negative effect? I just want to know is it worthy to invest the man/hour/money to clean those issues. As it involves decent development time. Is it common to have some of error/warning in most website that rank well? (e.g. I 've seem may big website have duplicate title/meta-desc on their currency variant page)0 -
Need some advice on changing website around.
I want to make some changes to my personal website and need a little help. I originally hired someone to create this site for me several years ago. Most of the site (including the home page) is html, but the blog part (/blog) is wordpress. The main site hasn't been updated in a few years (although the blog has been updated more often.) I'm not really targeting any keywords on my site (other than just my name) so I'm not really concerned about loosing any rank or anything. I"m just curious what the best route to go should be. Would you suggest keeping the site's format "as is" so to speak? (make the front page a landing page so to speak that links to my various sites, a little bio, etc; keep an about page, contact, /blog, etc...? ) Or should I go the more traditional route and make the blog the front page? Another route is to someone keep the most recent (1,2,or 3) posts on the home page (landing page) and sort of make it a landing page / recent post page. It's sort of like that now, only the main page just links to the most recent post instead of actually adding the post, if that makes sense. There maybe some themes out there that would work well, so feel free to suggest one if you know one that would work. I think the Aboutme.com pages look pretty nice and will let you use a custom domain, but I'm not sure. (And it's not a complete theme, so not sure that's the best option.) Thanks!
Web Design | | NoahsDad0 -
How to make AJAX/javascript website more seo friendly?
I have a website that is heavy on AJAX. I need recommendations on how to add content as well as other on page ompitizations. The website is a luxury brand for 6 resorts, each with their own subfolder. The website is http://me.graficode.com/preprod/.
Web Design | | Melia0