Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I am Using <noscript>in All Webpage and google not Crawl my site automatically any solution</noscript>
| |
Web Design | | ahtisham2018
| | <noscript></span></td> </tr> <tr> <td class="line-number"> </td> <td class="line-content"><meta http-equiv="refresh" content="0;url=errorPages/content-blocked.jsp?reason=js"></td> </tr> <tr> <td class="line-number"> </td> <td class="line-content"><span class="html-tag"></noscript> | and Please tell me effect on seo or not1 -
Is The HREF Link "Title" Tag Needed on Mobile Websites?
Hello To Those Who Are Wiser Than I, I am wondering if the href link "title" tag is needed, or serves any purpose, on mobile websites? Also, does it effect SEO in any way? I ask because generally the href link title tag provides more information to the user when they scroll their mouse over the link - but this action does not happen on mobile! Users have no mouse and thus no extra information would be displayed. I'm really wondering if it still matters for SEO purposes on mobile though. -The UnEnlightened
Web Design | | Stew2220 -
What are the downsides and/or challenges to putting page paths (www.example.com/pagepath) on a different server?
Hi, Our company is organized into three different segments and our development team recently needed to switch a portion of the business to subdomain because they wanted to move to a different server platform. We are now seeing the impact of moving this segment of the business to a subdomain on the main domain. SEO is hurting and our MOZ score has dropped significantly. One fix they are debating is moving everything back to one domain, but place segments of the business on different page paths and hosting specific paths on different servers. I.e. the main domain could be www.example.com hosted in one location and then www.example.com/segment1 would be hosted on a different server. They are hoping to accomplish this using some sort of proxy/caching redirection solution. The goal of this change would be to recapture our domain strength. Is this something that is a good option or no? If not, what are the challenges and issues you see arising from doing something like that as I don't know of any other site set up like this. Thanks in advance.
Web Design | | bradgreene0 -
Question #2: All of my INTERNAL links in OSE are being indexed from http://www.e.com/default.asp, and all my EXTERNAL links are linked to http://www.e.com/ am I getting a fraction of the link juice because of that?????
Hey guys, sorry for the really long question, but it appears that I am losing between 50 and 75 % of my link juice to my internal pages. In OSE all main category links (left sidebar) are being indexed from the URL that includes default.asp, even though NONE of my external links include that: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2FSulky-Thread-s%2F78.htm If you check the PA for http://www.uncommonthread.com/: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2F You see that it is practically double the PA of http://www.uncommonthread.com/default.asp: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2FDefault.asp **Also, non of my internal menu links are being indexed. ** Look at the menu on this page: http://www.uncommonthread.com/Sulky-Thread-s/78.htm and then look at the OSE information here for the "invisible thread" item from the menu on the page above^^^: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2FSulky-monofilament-s%2F54.htm Thanks SOOO much! Pre-thumbs and thanks to anyone that can lend me a seconds worth of advice! Thanks again for your time, Tyler A.
Web Design | | TylerAbernethy0 -
Does disabling the "View Source" functionality prevent Google from crawling a website?
I know Google uses a lot of variables when crawling a website. I wasn't sure if disabling the "View Source" option hindered anything.
Web Design | | innovationsimple0 -
Landing Page/Home Page issues
Hi. I was speaking with my designer last night (we are setting up a new website) and we were discussing the design of our homepage, now the designer said he wanted the first page of the website to be a sort of landing page page were the visitor has to click and enter, im sure everyone has all come across these before. However, I am concerned as to the SEO implications of this? Any help guys?
Web Design | | CompleteOffice0 -
Does listing my customer's address, phone number, and a contact form on "every page" count as duplicate content that they'd be penalized for?
I work with small local businesses (like Tree Farms, Feed Stores, Counselors, etc) doing web design, seo, etc. I encourage them to have their contact information visible at all times on their websites. I'm also delving into the world of contact forms. I want to have this info on every page - is this detrimental? Here's an example: http://www.trinityescape.net/marriage-couples-counselors-therapy-clermont-florida/ Thank you!
Web Design | | mikjgens1 -
How does the "first link" rule work with the "reasonable surfer patent" when it comes to the main navigation for a website?
In trying to figure out navigation for a new website, I am struggling with the first link rule vs. the reasonable surfer patent where the first link rule implies that Google "counts" the first link to a page including navigation, and the reasonable surfer patent that implies that navigation links carry less weight than body copy links. What is the best solution for creating main navigation so that it doesn't take away from the body copy links?
Web Design | | cindyt-170380
