Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do SEOs really need to care about trend in increase of voice search?
Hi all, Just browsing through the articles like "SEO trends in 2108"....I can see everyone mention about increase in voice search and attributing same to the SEO at top. But there is very less information on how to optimise for voice search....May be, they don't know what SEOs have to do exactly with voice search. Basically the input method is going to change on search and else remains same. This must be leading to increase of long tail searches and more search snippets to give an instant voice answer. What else has to be optimised to favoured in voice search? Thanks
Web Design | | vtmoz2 -
Using Multiple links/names for the same product?
I am being asked to change these product links on the home page: Home/Condo
Web Design | | RoxBrock
Watercraft/Boat to Home
Condo
Watercraft
Boat (Along with several other product links) How does this affect the customer experience/usability, and SEO? Is it a good idea or is it confusing? Thank you.0 -
New To SEO Management, I just want to double check that my idea will work.
I am new to SEO management. I had a 3 month SEO copy writing internship and a 5 month SEO temp job. In both I mostly wrote copy, but I've been teaching myself SEO on the side, I became Google certified. I ended up getting a telemarketing job and somehow the conversation of SEO came up and I winded up managing their SEO for 12 dollars an hour. They say that every lead generated from the website that turns into a sale will be worth 10 dollars and if and when the sales exceed my paycheck I will starting making commission so long as it stays above my hourly. SEO is very fun and this is like my dream job. They are leaving the planning 100% up to me and I want to make sure that what I am doing will work. My plan is as follows: Part 1: Page Authority via backlinks and social media We are health care brokers and my boss, the owner has a lot of contact. He is talking with large unions like, "The Teamsters," and large company retirment groups like, "Blue flame," which is apparently in some way connected to DTE or GE. Long story short, I am trying to get him to convince them to give us a back link to our main page. He also has a ton of clients that own companies. This is good because they may be persuaded to give us backlinks too. In addition, the tech guy thinks he can implement something where we can get a google +1, facebooks likes/shares, twitter likes and shares and pintrest pin it's that would be a part of an email that we send to people within the list of 12,000 clients. From what I can see, from the client base and the people we are working with we should be able to raise the page authority substantially despite the fact that the site is only a few months old and is not yet out of the sand box. I have been slowly picking off each error with SEO MOZ's website crawling. Part 2: Making a Insurance Jargon Dictionary Guide For The Tri-purpose of gathering traffic, proving our professionalism and helping people understand semi-complex insurance jargon. I could build these 2-3 keywords would be addressed per page and they would be defined in a way to help people looking for terms understand them, while simultaneously netting a strong keyword density and a strong page. I think as far as I can tell there are no issues. Part 3: The dictionary pages will pull in new traffic and the home page will receive links and distribute link juice to the sub-pages. This subpages will guide traffic back to the main page with no-follow links to direct people from the unique termed landing pages to the home page for insurance processing. As far as I can tell my logic is solid and on paper this should work. Am I missing anything (like key details, flaws in my plan)?
Web Design | | Tediscool0 -
Web development - License CSS/Markup/Code
In development of a website, is it typical for the developer to retain rights to the CSS, Markup and other Coding? If so, why is this done?
Web Design | | DemiGR0 -
Does it do harm if you add a rel="canonical" tag on a page that doesn't need it?
If a page is clearly unique and there is obviously no canonical tag needed, does it hurt anything if one has been added?
Web Design | | jaychow0 -
Need help to implement microdata/microformat for ecommerce site
**Can somebody please help me to implement microdata/microformats codes for our ecommerce product pages? **
Web Design | | EastEssence22
Please guide me if you have some CSS example for the same. Thanks.0 -
CSS styling help needed
I'm hoping that to someone experienced this is a quick fix, but it may be a pain and a rat's nest of code. Five or so years ago, I designed my brother's website at https://www.argentdata.com/index.html. I built it off of the Mollio theme (http://www.mollio.org/), and used templates in Dreamweaver, so it's not in a CMS. It's not a work of art, and there are some issues with it, but it's served him fairly well. Longer-term it needs a redesign, but right now I'm asking about just one aspect of the existing design. There are two parts to the site - the plain HTML site, and the osCommerce store that's under the purchase tab. If you go to http://www.argentdata.com/catalog/ you'll see that the osCommerce store is full width, while the navigation header copied over from the main site is 1200 pixels wide and centered. If you're on a wide monitor, it looks weird. What he'd like to do: Make the main site align to the left, both the content and navigation, and make the navigation left aligned for the osCommerce portion. He figures this is the easier way to do things, rather than try to muck with osCommerce CSS (that also has some merged CSS from the Mollio theme) and get it to center. Does anyone have a fairly simple solution that can make the navigation bar look good on osCommerce? Either the above of making everything left aligned, or some other solution simple we haven't thought of? Thanks!
Web Design | | KeriMorgret0 -
Is it OK to 301 Redirect http://homepage.com to http://blog.homepage.com?
I don't have a homepage built yet, and I suspect that my blog will be my primary SEO draw in the long term, so I'd like to do a 301 redirect (for the techies, I'm doing it in Django with http.HttpResponsePermanentRedirect ) I just wanted to confirm that this wouldn't cause any problems as Google gets used to my page (it's only been up for a few days) Thanks! -matt
Web Design | | 49wetnoodles0