Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need suggestion: What is the best internal linking structure for our website to gain in SEO & UX too?
Hi all, We have 3 different editions of our product we are selling with 20 features. 1st edition & 2nd edition comes with 15 features in which 10 are common in each edition. 3rd edition comes with all 20 features. Now what's the best way to interlink and show the navigational menu to highlight 3 editions and features as well? Much appreciated if some one refer me a website with such structure. Thanks
Web Design | | vtmoz0 -
Should I Kill the Old Domain or Work Through the Redirect?
Our IT department wasn't able to create a new directory on the current domain name for whatever reason and so we had to create a new domain name called ww2.domain.com to build the new site. So now we have the new site up and appartly some PDFs and pages are being directed to the from the old site. www.domain.com but 10,000 pages /PDFs are still indexed in Google and are not redirected. So when you open the page you get the old www.domain.com instead of it redirecting to ww2.domain.com. It's sort of a mess! My question is can we just kill the old domain name and move the ww2.domain.com back to the old domain? We also want to do away with the ww2.domain.com and go back to www.domain.com. I know it's confusing as heck! What would you recommend?
Web Design | | Eagle-ABS0 -
Infinite Scrolling & "Long Scrolling" same or different??
Can anyone please confirm for me the difference if any, between site design that incorporates Long Scrolling and Infinite Scroll? I was told (by an unnamed source) these were different designs and that "long scrolling" is better for SEO . However, in all my research I am unable to prove there is any difference between the two. I understand Infinite scroll may include Ajax, but does that mean Long Scrolling does not? If anyone has any references or can supply any further education here, I'd appreciate it! Thanks!
Web Design | | ACNINTERACTIVE0 -
Http://www.domain.com/services/city or http://www.domain.com/city
Hi, I have a website created on a WordPress Platform. My site is in the "Service" industry and we perform the same service for many different "cities." My question pertains to the Navigation Bar and SEO. Is it better to have a "Service" tab on the navigation bar - that has sub navigation that lists all the cities on a drop down menu. When this happens the URL string looks like http://www.domain.com/service/city The other choice would be to create individual TABS on the NAV bar, by doing this the URL string would look - http://www.domain.com/city . I could be wrong, but I am assuming the http://www.domain.com/city is better than http://www.domain.com/services/city for SEO purposes, ....if I am correct is there a way to make SUB Menu URLS appear as http://www.domain.com/city ? Any input as always would be appreciated best regards, Jimmy
Web Design | | jimmy02251 -
Multiple Sites, multiple locations similar / duplicate content
I am working with a business that wants to rank in local searches around the country for the same service. So they have websites such as OURSITE-chicago.com and OURSITE-seattle.com -- All of these sites are selling the same services, but with small variations in each state due to different legal standards in the state. The current strategy is to put up similar "local" websites with all the same content. So the bottom line is that we have a few different sites with the same content. The business wants to go national and is planning a different website for each location. In my opinion the duplicate content is a real problem. Unfortunately the nature of the service makes it so that there aren't many ways to say the same thing on each site 50 times without duplicate content. Rewriting content for each state seems like a daunting task when you have 70+ pages per site. So, from an SEO standpoint we have considered: Using the canonocalization tag on all but the central site... I think this would hurt all of the websites SERPs because none will have unique content. Having a central site with directories OURSITE.com/chicago -- but this creates a problem because we need to link back to the relevant content in the main site and ALSO have the unique "Chicago" content easily accessable to Chicago users while having Seattle users able to access their Seattle data. The best way we thought to do this was using a frame with a universal menu and a unique state based menu... Also not a good option because of frames will also hurt SEO. Rewrite all the same content 50 times. You can see why none of these are desirable options. But I know that plenty of websites have "state maps" on their main site. Is there a way to accomplish this in a way that doesn't make our copywriter want to kill us?
Web Design | | SysAdmin190 -
Link Pages/Directory
Hello, What is best practise for dealing with alot of links. I was thinking of breaking them download to alphabet pages i.e. all A on one page etc... BUT should I then make the links clickable on this list OR that they load to a sub company page which has a clickable link to there website.
Web Design | | JohnW-UK0 -
Wordpress/ Insert Tables/ SEO
I'm using Wordpress to create websites and blogs. I have limited (non-existent) HTML Coding knowledge. I'm looking to insert tables within my pages with information. Inside of these tables I want certain names to link to another page with more specific information about that name. I'm using a plugin called "WP Tables Reloaded" it simple helps you to create aesthetically pleasing tables without needing to know HTML Code or CSS. The issue is... when you create this table and insert it to the post, the only thing that shows on the sites back-end page is the table I.D. and the only thing that shows in the HTML is the tables I.D. It looks like this... [table id=2 /] I don't think search engines will be able to crawl this table, thus I won't be receiving any credit for the links being used within the table. Am I right about this?
Web Design | | AndySolo0 -
Optimzing a new ecommerce site, Need help with URL
Hi We are putting up a new ecommerce website and for product description, our tech team indicates that they must have the skun numbers in the URL. Which one of the following URL structure do you find the most SEO freindly? 1. http://www.Site.com/SKUNumber/ProductDescription/ or 2. http://www.Site.com/ProductDescription/SKUNumber/ My personal opinion is that most relevant content should be on load page so I like option 1. Thanks
Web Design | | CookingCom0