Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do we need both an .XML Sitemap and a .aspx sitemap?
Hi Mozers, We recently switched servers and it came to my attention that we have two sitemaps a XML version of the sitemap and a .aspx version of the sitemap. This came to light as the .aspx version of the sitemap is causing the site to come to a screeching halt as it has some complex code and lists over 80,000 products. My question is do we need both versions of the sitemap? My understanding is that the XML version is for Search Engine bots and the .aspx version is for customers. I can't imagine that anyone is using our .aspx version as it is basically a page with 80,000 links and it's buried away on the site, so we were hoping to kill off the .aspx version of the sitemap and keep the .xml version for Search Engine Bots. I wanted to check here first to make sure we did not any negative search engine implications. Any help would be most appreciated. Thanks so much! Patrick
Web Design | | gatorpool0 -
Rich snippets tool/ schemer.org update?
We have a domain in the netherlands that shows stars (rich snippets) in adwords and in the organic search results. As we do the the in rich snippets test tool we see some errors as shown in the attachments, the errors are shown in review aggregate - but stars are stil shown in paid and organic search. Do we need to follow-up these errors? Do they influence ranking - or the stars in the near future? Z0fCKRW
Web Design | | remkoallertz1 -
Web Hosting and CDN for Wordpress Site Load Speed - Suggestions Needed
We all know that website load speed is more important than ever. While I love the look and feel of parallax and Wordpress, I want to do everything I can to keep the load speed down. I see a lot of conflicting information regarding web hosting services, CDN services and other service (Cloudflare for example). I am looking to hear from those with their own experiences to let me know what they think is the ideal setup for a parallax Wordpress site is as far as which services to use, including: 1. Web Hosting
Web Design | | Gauge123
2. CDN
3. Any other service or product that would help to provide and extremely fast site load time. Thank you!0 -
How to verify http://bizdetox.com for google webmaster tools
Hey guys i tried to to make a Preferred Domain choice in webmaster tools, but it is not allowing me to save my choice bec its asking me to verify that i own http://bizdetox.com How do i go about doing that and what are the steps I have already verified www.bizdetox.com
Web Design | | BizDetox0 -
Should I nofollow my blog "read more" links?
I have a standard blog index page with entries formatted as such: Blog title Blog excerpt read more link The blog title is linked to the blog post (duh). The "read more" link is also linked to the same blog post, but I put a rel="nofollow" tag on that link because I don't want the SE's thinking "read more" is relevant anchor text. Now, is what I'm doing right? Won't having the read more link as nofollow result in some sort of conflict considering the blog title link is set to follow? Will a nofollow link have a cascading effect on any matching "follow" link? I've frequently read that "read more" should be nofollowed, but I've never once seen anyone address the conflict of having a follow and nofollow link on the same page pointing to the same link. Which one wins? Thanks.
Web Design | | bluekite770 -
Word Press Seo Errors/ Questions
Hi my name is Tina I am new here I hope you guys can help me out. I thought building my new site with Word Press was going to simplify things, however I have a ton of errors, and I am not sure what they are, or how to fix them. I am hoping someone could share with me a solution for these errors. I have 28 rel=canonical errors, I am not sure what this means, I understand it to mean my pages are similar, and this is to set a heirarchy between my pages. Please correct me if I am wrong. If I am correct would this be necessary to add if my main keyword was "widgets" and my home page was optimized for "widgets" and my next page was "blue widgets" and so on. While my pages are similar they are all optimized for different versions of my main keyword some using long tail keywords. Do you know of a plugin that can help solve this problem? Also does anyone have a plugin they recommend for G+ my G+ authorship verification is causing an error as well? I am using Head Space 2 I have used this seo plugin numerous times with great success it has been my favorite seo plugin. However, we have a portfolio that shows our clients websites, and on those pages Head Space will not let me enter a description tag. What plug in do you guys recommend with more control over each page? Another interesting issue is on one of our pages I optimized it for our Canadian clients, and now every page has been listed in Google.ca for the keywords it should have on Google.com. We are listed on Google maps, verified in Google places, and our address is on the site so they know we're from the USA however, the majority of our keywords are only listed in Google.ca. We're on page one for all of them, we are in the top three on most of them so that's not bad, but we want to be listed in Google.com as well. Any suggestions on this?
Web Design | | TinaGammon1 -
Hom much does getting a mobile website improve the "mobile ranking"?
There's been speculation about Google totally not ranking sites that are not mobile, but as far as I can see many sites rank THE SAME on mobile devices as on regular stationary PCs/lap-tops. I had hoped for a radically improved ranking once a mobile site was built. Or is it just that it takes a TON of time for it to go through and then it gets better or is it like with regular sites that a new mobile site on a sub-domain is parked in some form of mobile-sand-box? Also, does putting a mobile site on a ranking domain, as subdomain help, e.g. m.testingsite.com instead of testingsite.mobi ? Thanks.
Web Design | | yvonneq0 -
How WP Themes work with Navigation Structure for SEO and JQuery Headers?
I am trying to find the best WP Theme for our company. I noticed most of them do not offer a left hand side navigation on the home pages, and usually are on the right side on the inside pages. I always thought that Home Page links were very important for SEO. Currenly we have a left drop down navigation with all of our product catagories, keyword optimized. The structure follows for all the pages. Is this not as important to Search Engines anymore? Is it better to have a products link, to all the products and then the inside pages, have just a navigation bar, for that particular catagory? This seems to be very common on all the templates i am seeing. I also noticed, and really like the JQueary Tabs. I would use this for displaying, PDFs and Specifications Charts. Also, some home page images are using a jquery slider with some text, linking to a page. Is Jquery the new javascript and do search engines see what is in the code? I also noticed they all have footers that have links and some other information. Is this a SEO must have?
Web Design | | hfranz0