Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is Prerender.io/React going to negatively impact our SEO efforts?
On any page on the site (https://theadventurepeople.com/), the same short code appears. Having investigated Google index pages, Google's cache and Fetch & Render, it does look like Google can view the content and index it, but we're not 100% convinced. Background technical information from the web developer: The website is a single page application built using React. The site is setup with Prerender: https://prerender.io/ (which renders the javascript in a browser, saves the static HTML, and returns that to crawlers). Is Prerender.io/React going to negatively impact our SEO efforts?
Web Design | | Wagada0 -
Responsive Site has "Not Found" Errors for mobile/ and m/ in Google Search Console
We have recently launched a new responsive website for a client and have noticed 2 "Not Found" errors within Google Search Console for /mobile and /m Both these URLs are not linked from anywhere within the site. However Google is reporting them as being linked from the homepage. This is not the first site we have seen in which Google has reported this error, however the other site was not a mobile friendly site. My thoughts are to 301 them back to the Homepage. Anybody else have any thoughts on this? or have recently received the same errors?
Web Design | | JustinTaylor881 -
Advice needed: Google crawling for single page applicartions with java script
Hi Moz community,we have a single page application (enjoywishlist.com) with a lot of content in java script light boxes. There is a lot of valuable content embedded but google can not crawl the content and we can missing out on some opportunities as a result. I was wondering if someone was able to solve a similar issue (besides moving the content from the java script to the HTML body). There appears to be a few services sprouting up to handle single page applications and crawling in google.http://getseojs.com/https://prerender.io/Did anyone use these services? Some feedback would be much appreciated!ThanksAndreas
Web Design | | AndreasD0 -
Question #2: All of my INTERNAL links in OSE are being indexed from http://www.e.com/default.asp, and all my EXTERNAL links are linked to http://www.e.com/ am I getting a fraction of the link juice because of that?????
Hey guys, sorry for the really long question, but it appears that I am losing between 50 and 75 % of my link juice to my internal pages. In OSE all main category links (left sidebar) are being indexed from the URL that includes default.asp, even though NONE of my external links include that: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2FSulky-Thread-s%2F78.htm If you check the PA for http://www.uncommonthread.com/: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2F You see that it is practically double the PA of http://www.uncommonthread.com/default.asp: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2FDefault.asp **Also, non of my internal menu links are being indexed. ** Look at the menu on this page: http://www.uncommonthread.com/Sulky-Thread-s/78.htm and then look at the OSE information here for the "invisible thread" item from the menu on the page above^^^: http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.uncommonthread.com%2FSulky-monofilament-s%2F54.htm Thanks SOOO much! Pre-thumbs and thanks to anyone that can lend me a seconds worth of advice! Thanks again for your time, Tyler A.
Web Design | | TylerAbernethy0 -
Yes or No for Ampersand "&" in SEO URLs
Hi Mozzers I would like to know how crawlers see the ampersand (& or &) in your URLs and if Google frown upon this or not? As far as I know they purely recognise this as "and" is this correct and is there any best practice for implementing this, as I know a lot of people complained before about & in links and that it is better to use it as &, but this is not on links, this is on URLs. Reason for this is that we looking to move onto an ASP.Net MVC framework (any suggestions for a different framework are welcome, we still just planning out future development) and in order to make use of the filter options we have on our site we need a parameter to indicate the difference on a routing level (routing sends to controller, controller sends to model, model sends to controller and controller sends to view < this is pattern of a request that comes in on the framework we will be using). I already have -'s and /'s in the URLs (which is for my SEO structuring) so these syntax can't be used for identifying filters the user clicks or uses to define their search as it will create a complete mess in the system. Now we looking at & to say; OK, when a user lands on /accommodation and they selects De Kelders (which is a destination in our area) the page will be /accommodation/de-kelders on this page they can define their search further to say they are looking for 5 star accommodation and it should be close to the beach, this is where the routing needs some guidance and we looking to have it as follow: /accommodation/de-kelders/5-star&close-to-the-beach. Now, does the "&" get identified by search engines on a URL level as "and" and does this cause any issues with crawling or indexation or would it be best to look at another solution? Thanks, Chris Captivate
Web Design | | DROIDSTERS0 -
Site Change / CMS Change
Hi Guys I have a very good client with whom we have been working with for over 2 years. When we 1st arrived, their website was built in Wordpress. During our SEO, the dleted the site and changed it to Drupal over night. Youc an imagin i went balistic and its been a horrid year in getting all the rrors down and redirects etc. Finally done I have just been informed that they will be deleting the website from Drupal to Joomla again! This is crazy as we have finally fix all the rrors and are driving quality traffic to the site. Now its back to square one... What do i do? ps the change from Drupal to Joomla is due to MYSQL issues
Web Design | | stefanok0 -
Optimzing a new ecommerce site, Need help with URL
Hi We are putting up a new ecommerce website and for product description, our tech team indicates that they must have the skun numbers in the URL. Which one of the following URL structure do you find the most SEO freindly? 1. http://www.Site.com/SKUNumber/ProductDescription/ or 2. http://www.Site.com/ProductDescription/SKUNumber/ My personal opinion is that most relevant content should be on load page so I like option 1. Thanks
Web Design | | CookingCom0 -
Hosting/design company that is both cheap and has a nice partner package. Any ideas?
I need to signon with a hosting/design company that is both cheap and has a nice partner package. Any ideas?
Web Design | | christinarule0