Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google is a definition engine / Question answering engine
But what do you do to rank when you sell a product and people don't have questions and none of the companies that rank on your keywords answers questions... Let me give you a few example. Title tag for example : People want to know what is it and then have question about it What is the maximum number of characters ? How do you write a good one ? Etc... How to remove stains is another example : Please want to know how to remove all the different types of stains, (ketchup, grease etc..) But what about when you are a online business and want to sell usb keys , rent bicycles https://www.spinlister.com or even sell a software to do A/B testing on your website https://bit.ly/2a6cBuF Can someone explain me how those people mentioned rank without giving definition or answering questions... because according to me they don't do any of that on the pages that I mentioned. I look forward to your reply. Thank you,
Web Design | | seoanalytics0 -
Privacy Policy: index it/? And where to place it?
Hi Everyone, Two questions, first: should you allow google to index your privacy policy? Second: for a service based site (not e-commerce, not selling anything) should you put the policy in the footer so it's site wide or just on the "contact us" form page? Best, Ruben
Web Design | | KempRugeLawGroup0 -
Wordpress Plugin To Capture Form Completion Data, Before Visitor Hits "Submit"?
Greetings MOZ Community: Visitors frequently start to enter contact information in the forms on our website, but then chicken out and don't hit the submit button. I noticed this watching the recordings of visitor web site visits using Mouse Flow. Is there a Wordpress Plugin that would allow us to capture data entered in forms, where the visitor does not finally hit the "submit" button? Obviously this would be very, very valuable as this scenario occurs in one out of three or four instances. Thanks!!!
Web Design | | Kingalan1
Alan0 -
A/B Testing.. Are you doing? how is it been? What do you think would be the best path for who is starting now?
Hey Mozers, One of my 2014 resolutions is to start doing A/B Testing, so far I have been following "best practices" and "common sense" when comes to website design, but I would like to go above and beyond. I was hoping a could get a few tips some of you that are already doing A/B testing. How is it been? Do you see a great ROI? What do you think would be best path for who is starting now? Any book or links you would recommend? Thanks
Web Design | | Felip30 -
Google result showing old Meta Title / Description even though page view source shows new info.
Hey guys! I'm struggling with why Google is ignoring my Meta Title / Description. I made a pretty drastic change to both about a week ago and on the results it hasn't changed. I'm on first page with several keywords and I think this weird caching is hurting me on where I'm at on the page. Thoughts / Ideas?
Web Design | | curtis_williams0 -
Should I nofollow my blog "read more" links?
I have a standard blog index page with entries formatted as such: Blog title Blog excerpt read more link The blog title is linked to the blog post (duh). The "read more" link is also linked to the same blog post, but I put a rel="nofollow" tag on that link because I don't want the SE's thinking "read more" is relevant anchor text. Now, is what I'm doing right? Won't having the read more link as nofollow result in some sort of conflict considering the blog title link is set to follow? Will a nofollow link have a cascading effect on any matching "follow" link? I've frequently read that "read more" should be nofollowed, but I've never once seen anyone address the conflict of having a follow and nofollow link on the same page pointing to the same link. Which one wins? Thanks.
Web Design | | bluekite770 -
I need to buy a website template with seo focus - any tip? what technology is better?
Due to lack of resources, my company usually buys website templates and we adjust it internally. So we can always be with new designs, spending less money. With all this changes of google panda recently, I will change my website again. Usually we buy then from templatemonster, but some templates are far away from seo standards. Do you have tips where to buy than? Should I search for a specific technology or layout? Is this JS Animated website a goog choice?
Web Design | | Naghirniac0