Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do we still have this Page Rank / Link juice / Link equity? So this dilution concept?
Hi all, As per the traditional or standard SEO rules, we have this link juice and dilution concept. Many websites have changed their linking structure with this with the beleif "the more number of pages, the PR will get diluted". Then many websites avoided more number of pages from homepage to avoid link juice dilution. Even we followed same. But I just wonder it's still the same way Google handles websites and rankings as per the links. And many websites even avoid more number of 2nd tier/hierarchy pages to avoid link dilution. I have gone through our competitors where they been employing lot of top level pages like 2nd tier/hierarchy pages but still doing good at rankings. Please share your views and suggestions on this. Thanks
Web Design | | vtmoz0 -
Wordpress SEO/Tag plugin recommendation for sports (soccer) website
I own a Wordpress website which covers soccer in the DC MD VA area called DMV Soccer http://www.dmvsoccer.com/ We write weekly recaps where we tag a player who has scored a goal or performed well in a game. For each player, obviously, a tag is created. What I'm looking for is a plugin or solution that would allow me to tag a player, but also automatically assign a team to that player so that the team name and player's name are optimized on the individual player's tag page. So if I were to tag George Murphy on a recap, and I assign him to a team, let's say DC United. The tag page would have a title, something like: George Murphy Soccer Player for DC United and the meta description: George Murphy, soccer player from MD who players for DC United archives Or something similar, if that makes sense. Should I skip using tags and instead start assigning each player as a sub-category under each team? I'd like to try to avoid that, because not each category will be based on a player. Any suggestions in terms of existing plugins or other recommendations?
Web Design | | georgetsn1 -
Blog.yoursite.com or yoursite.com/blog/?
Rand made the following post in 2009. In July of 2013 I question whether this is still best practice. Obviously, much has changed in our industry in four-and-a-half years! "...if you're seeking to maximize your ranking ability for a given piece of content, it's my personal belief that you should, most of the time, keep it on 1 subdomain under 1 root domain (but feel free to use subfolders as it makes sense). Starting a blog? I almost always recommend yoursite.com/blog over blog.yoursite.com. Want to launch a new section of content? Use yoursite.com/newstuff rather than newstuff.yoursite.com." Source: http://moz.com/blog/understanding-root-domains-subdomains-vs-subfolders-microsites
Web Design | | JorgeUmana0 -
How to do a non-spammy "doorway page"?
Hi there, ISSUE: I have a client who wishes to use a "doorway" page, but not in a spammy way. He would like to have a nice crisp URL for use in ads/brochures. The page is strictly a landing page (just with a separate URL). DOORWAY/LANDING PAGE WILL BE: Non-spammy -- There will be no attempt to optimize the landing page/no attempt to get the page to rank. Strictly a vanity URL -- he likes the way a separate website looks in ads as opposed to a landing page on the existing website (i.e., www.websitename.com/landing page) WHAT I'M TRYING TO DO: I'm basically trying to figure out what the best things to do to protect his other sites (which are very high quality valuable sites which rank well) from getting punished. STEPS I'M CONSIDERING: Robots no follow Separate hosting server Different person's name on a private domain registration Adding additional pages, so it's not a 1-page "doorway" Many thanks in advance to anyone who would share their experience and help me protect my client in the best way possible. I've told him there are risks, but he still wants to go ahead. MC
Web Design | | marketingcupcake1 -
Should I nofollow my blog "read more" links?
I have a standard blog index page with entries formatted as such: Blog title Blog excerpt read more link The blog title is linked to the blog post (duh). The "read more" link is also linked to the same blog post, but I put a rel="nofollow" tag on that link because I don't want the SE's thinking "read more" is relevant anchor text. Now, is what I'm doing right? Won't having the read more link as nofollow result in some sort of conflict considering the blog title link is set to follow? Will a nofollow link have a cascading effect on any matching "follow" link? I've frequently read that "read more" should be nofollowed, but I've never once seen anyone address the conflict of having a follow and nofollow link on the same page pointing to the same link. Which one wins? Thanks.
Web Design | | bluekite770 -
Word Press Seo Errors/ Questions
Hi my name is Tina I am new here I hope you guys can help me out. I thought building my new site with Word Press was going to simplify things, however I have a ton of errors, and I am not sure what they are, or how to fix them. I am hoping someone could share with me a solution for these errors. I have 28 rel=canonical errors, I am not sure what this means, I understand it to mean my pages are similar, and this is to set a heirarchy between my pages. Please correct me if I am wrong. If I am correct would this be necessary to add if my main keyword was "widgets" and my home page was optimized for "widgets" and my next page was "blue widgets" and so on. While my pages are similar they are all optimized for different versions of my main keyword some using long tail keywords. Do you know of a plugin that can help solve this problem? Also does anyone have a plugin they recommend for G+ my G+ authorship verification is causing an error as well? I am using Head Space 2 I have used this seo plugin numerous times with great success it has been my favorite seo plugin. However, we have a portfolio that shows our clients websites, and on those pages Head Space will not let me enter a description tag. What plug in do you guys recommend with more control over each page? Another interesting issue is on one of our pages I optimized it for our Canadian clients, and now every page has been listed in Google.ca for the keywords it should have on Google.com. We are listed on Google maps, verified in Google places, and our address is on the site so they know we're from the USA however, the majority of our keywords are only listed in Google.ca. We're on page one for all of them, we are in the top three on most of them so that's not bad, but we want to be listed in Google.com as well. Any suggestions on this?
Web Design | | TinaGammon1 -
Having a new website build, what happens to my SEO work?
Hi, We are in the process of having a new website developed by a web agency. New domain, new business name etc... What is the best process to transfer from old domain to the new one? What happens to all my ranked URLs? What can re-directs do to help me? Will I loose all my link juice on highly ranked keyword domains? I ranked 1st for a brand now, wil lthis go? Any advice, help or hints would be great. Thanks Will
Web Design | | YNWA0 -
I NEED ADVICE: Need Web Software for customers to access backend
I have a prospect in the credit repair industry. Does anybody know of some software where the clients can log into their own accounts to see the progress? The feature I am looking for is like a Real Estate website where the brokers can create their own accounts and upload their own properties.
Web Design | | Francisco_Meza0