Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do we need to interlink homepage to the highest?
Hi, So our homepage is not even at top 5 pages interlinked most. I could see a ranking drop even when other pages overtaken homepage in internal linking. Homepage is the highest priority page we are expected to rank for our primary keyword. So, not linking homepage internally high but giving importance to other pages confuses Google and outrank homepage in rankings? Thanks
Web Design | | vtmoz0 -
Google result showing old Meta Title / Description even though page view source shows new info.
Hey guys! I'm struggling with why Google is ignoring my Meta Title / Description. I made a pretty drastic change to both about a week ago and on the results it hasn't changed. I'm on first page with several keywords and I think this weird caching is hurting me on where I'm at on the page. Thoughts / Ideas?
Web Design | | curtis_williams0 -
Does using role="heading" instead of H1 in HTML code affects SEO?
Does using role="heading" instead of affect SEO? http://www.w3.org/WAI/GL/wiki/Headings_using_role%3Dheading
Web Design | | LNEseo0 -
Having a new website build, what happens to my SEO work?
Hi, We are in the process of having a new website developed by a web agency. New domain, new business name etc... What is the best process to transfer from old domain to the new one? What happens to all my ranked URLs? What can re-directs do to help me? Will I loose all my link juice on highly ranked keyword domains? I ranked 1st for a brand now, wil lthis go? Any advice, help or hints would be great. Thanks Will
Web Design | | YNWA0 -
Need to rebuild client's flash website
I am working with their web designer and need to figure out a way to rebuild their site which is currently all in flash. I was wondering if there was a way to do this without spending a ton of time in completely re-doing the site from scratch.
Web Design | | awalker840 -
Link Pages/Directory
Hello, What is best practise for dealing with alot of links. I was thinking of breaking them download to alphabet pages i.e. all A on one page etc... BUT should I then make the links clickable on this list OR that they load to a sub company page which has a clickable link to there website.
Web Design | | JohnW-UK0 -
Wordpress/ Insert Tables/ SEO
I'm using Wordpress to create websites and blogs. I have limited (non-existent) HTML Coding knowledge. I'm looking to insert tables within my pages with information. Inside of these tables I want certain names to link to another page with more specific information about that name. I'm using a plugin called "WP Tables Reloaded" it simple helps you to create aesthetically pleasing tables without needing to know HTML Code or CSS. The issue is... when you create this table and insert it to the post, the only thing that shows on the sites back-end page is the table I.D. and the only thing that shows in the HTML is the tables I.D. It looks like this... [table id=2 /] I don't think search engines will be able to crawl this table, thus I won't be receiving any credit for the links being used within the table. Am I right about this?
Web Design | | AndySolo0 -
Setup of three major retail sites.. need advice.
I recently have taken a new position responsible for three large national retail sites which are all owned by one parent organization. Through a series of acquisitions, these three major brands have been brought under one umbrella and a brand consolidation is likely not to happen within the next 2-4 years. I have a number of questions I’m hoping to get some feedback on, but first a little more background is necessary. A year ago (before my time) the three sites were over-hauled, but were designed to use one common custom CMS and all of the navigation and nearly all the content is the same (with some exceptions, such as tags, url, etc.). All of the brands have identical products and services; however, each one services a different demographic in the US. The design was intended for ease of management, but is terrible for seo. Additionally, without the geographic reference, they all compete for the same keywords. They have now begun a very large ecommerce project utilizing an ATG platform. The initial direction is to use one platform for all three brands, but keep them on separate domains and with the use of basic switching, replace nominal content such as logos and references of the brands for each of the domains. I’m concerned with this approach and would like to hear your feedback.. When optimizing a page for one keyword set, are they likely to be filtered due to dup content? The argument that management has is that all three current sites rank very well for one keyword on all three sites. They feel it won’t be an issue due to this. One option, that is currently still available, is to tri-band one ecommerce site, but it would have to be on an entirely new domain. The other three domains are very well established and are PR6s. Management, and even I, is afraid to abandon these other domains, but having a single domain would allow us to have unique content and really leverage all efforts to one domain. Thoughts? Any knowledge or thoughts what kind of impact having three domains on one ATG platform will be? Thanks much! John If you feel it will help, please message me and I can share the urls... Also, how would you handle a company blog in this case?
Web Design | | kavaliauskas0