Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Magento 2.1 Multi Store / SEO
This is quite technical but I'm hoping a Magento expert can clear this up for me. Currently my company has two websites on separate Opencart platforms. What I'm doing now is building a Magento website and using the multi store function as well as a few modules to combine the two sites, the aim being that the link juice is shared and I can focus my SEO efforts on the one site instead of two, thus reducing my workload while maintaining the benefits. This is the intended layout: www.domain.com www.domain.com/us I have created a sub-folder (not a subdomain) as this seems to be the best way to share link juice between the new, combined sites (as well as 301s from the old, redundant site). At the moment I have created 2 separate websites, stores and store views (see attached) and have configured it according to the Magento guide, so I know that technically this is correct but I need to make sure that I have done it correctly in relation to SEO. Is the sub-folder set up correctly for instance? Currently the only files to populate that sub-folder are a htaccess, error log and index.php (see attached). Also, is there anything I could be missing in relation to SEO within the parameters of what I am trying to achieve? Additionally, only one store view appears in the "change store view" section of the home page. This is causing me to question if I have set it up correctly, because I had assumed both store views would appear even if they were under different websites (attached). OR do I simply use the same website and create two stores and store views? Do I also need to create a separate database for each website/store/store view? I would very much appreciate if someone could help out here. Thank you. In1Gi7t pyfM03y nUQoMz1
Web Design | | moon-boots0 -
Homepage Gateway for Website Divisions (Residential / Commercial) Bad for SEO?
When a website offers multiple divisions for products and services would it be a good SEO practice to implement something as Culligan does by having their homepage be a sort of gateway to the different divisions of site (home, office, commercial, industrial)? 2DaYz
Web Design | | m-johnson0 -
Multiple websites for different service areas/business functions?
I'm wondering what the implications are for having multiple domains for different service areas of a company? I realize having multiple domains for one company can be troublesome because of the possibility of duplicate content, keyword cannibalization, and linkbuilding to multiple domains. But when the domains are for very different service offerings/unique business functions that each serve their own purpose (and have different positionings), is there a downside to having more than one domain? Any thoughts would be appreciated!
Web Design | | KevinBloom0 -
Google result showing old Meta Title / Description even though page view source shows new info.
Hey guys! I'm struggling with why Google is ignoring my Meta Title / Description. I made a pretty drastic change to both about a week ago and on the results it hasn't changed. I'm on first page with several keywords and I think this weird caching is hurting me on where I'm at on the page. Thoughts / Ideas?
Web Design | | curtis_williams0 -
How does using a CMS (i.e. Wordpress/Drupal) affect backlinks and SEO?
So I need to build a website with over 100 pages in it. Elements of the design will probably be moved around and or tested so I need to use a CMS. It's pretty much a review site so while the content will remain static I'd like to employ A/B testing to mess with conversion rates. Wordpress has a plugin for that even. So I'm just wondering, since CMS pages are pretty much created on spot and not retrieved from a library, how this affects backlinks and anchor text? How exactly does the external website point to yours if the URL is dynamically generated? Or am I misunderstanding something? Please recommend any extra resources as well if you can.
Web Design | | seochump0 -
Need some advice on changing website around.
I want to make some changes to my personal website and need a little help. I originally hired someone to create this site for me several years ago. Most of the site (including the home page) is html, but the blog part (/blog) is wordpress. The main site hasn't been updated in a few years (although the blog has been updated more often.) I'm not really targeting any keywords on my site (other than just my name) so I'm not really concerned about loosing any rank or anything. I"m just curious what the best route to go should be. Would you suggest keeping the site's format "as is" so to speak? (make the front page a landing page so to speak that links to my various sites, a little bio, etc; keep an about page, contact, /blog, etc...? ) Or should I go the more traditional route and make the blog the front page? Another route is to someone keep the most recent (1,2,or 3) posts on the home page (landing page) and sort of make it a landing page / recent post page. It's sort of like that now, only the main page just links to the most recent post instead of actually adding the post, if that makes sense. There maybe some themes out there that would work well, so feel free to suggest one if you know one that would work. I think the Aboutme.com pages look pretty nice and will let you use a custom domain, but I'm not sure. (And it's not a complete theme, so not sure that's the best option.) Thanks!
Web Design | | NoahsDad0 -
Wordpress/ Insert Tables/ SEO
I'm using Wordpress to create websites and blogs. I have limited (non-existent) HTML Coding knowledge. I'm looking to insert tables within my pages with information. Inside of these tables I want certain names to link to another page with more specific information about that name. I'm using a plugin called "WP Tables Reloaded" it simple helps you to create aesthetically pleasing tables without needing to know HTML Code or CSS. The issue is... when you create this table and insert it to the post, the only thing that shows on the sites back-end page is the table I.D. and the only thing that shows in the HTML is the tables I.D. It looks like this... [table id=2 /] I don't think search engines will be able to crawl this table, thus I won't be receiving any credit for the links being used within the table. Am I right about this?
Web Design | | AndySolo0