Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What to do with blog pagination / categories
On our new website we are going to have a blog that is custom built. However, you can browse the general blog with pagination. You can also nest into the categories or by month. Doing each will have a different link: Category: www.mywebsite.com/blog/web-guides/Month: www.mywebsite.com/blog/2014/january/ My question is what is the best thing to do with these pages. Rel Canonical them to the blog home? Or nofollow/noindex them? Will Google not see these as duplicate content pages?
Web Design | | vortexuk0 -
Web Hosting and CDN for Wordpress Site Load Speed - Suggestions Needed
We all know that website load speed is more important than ever. While I love the look and feel of parallax and Wordpress, I want to do everything I can to keep the load speed down. I see a lot of conflicting information regarding web hosting services, CDN services and other service (Cloudflare for example). I am looking to hear from those with their own experiences to let me know what they think is the ideal setup for a parallax Wordpress site is as far as which services to use, including: 1. Web Hosting
Web Design | | Gauge123
2. CDN
3. Any other service or product that would help to provide and extremely fast site load time. Thank you!0 -
Web development - License CSS/Markup/Code
In development of a website, is it typical for the developer to retain rights to the CSS, Markup and other Coding? If so, why is this done?
Web Design | | DemiGR0 -
Too Many On Page Links, rel="nofollow" and rel="external"
Hi, Though similar to other questions on here I haven't found any other examples of sites in the same position as mine. It's an e-commerce site for mobile phones that has product pages for each phone we sell. Each tariff that is available on each phone links through to the checkout/transfer page on the respective mobile phone network. Therefore when the networks offer 62 different tariffs that are available on a single phone that means we automatically start with 62 on page links that helps to quickly tip us over the 100 link threshold. Currently, we mark these up as rel="external" but I'm wondering if there isn't a better way to help the situation and prevent us being penalised for having too many links on page so: Can/should we mark these up as rel="nofollow" instead of, or as well as, rel="external"? Is it inherently a problem from a technical SEO point of view? Does anyone have any similar experiences or examples that might help myself or others? As always, any help or advice would be much appreciated 🙂
Web Design | | Tinhat0 -
Having a new website build, what happens to my SEO work?
Hi, We are in the process of having a new website developed by a web agency. New domain, new business name etc... What is the best process to transfer from old domain to the new one? What happens to all my ranked URLs? What can re-directs do to help me? Will I loose all my link juice on highly ranked keyword domains? I ranked 1st for a brand now, wil lthis go? Any advice, help or hints would be great. Thanks Will
Web Design | | YNWA0 -
Does listing my customer's address, phone number, and a contact form on "every page" count as duplicate content that they'd be penalized for?
I work with small local businesses (like Tree Farms, Feed Stores, Counselors, etc) doing web design, seo, etc. I encourage them to have their contact information visible at all times on their websites. I'm also delving into the world of contact forms. I want to have this info on every page - is this detrimental? Here's an example: http://www.trinityescape.net/marriage-couples-counselors-therapy-clermont-florida/ Thank you!
Web Design | | mikjgens1 -
Should /dev folder be blocked?
I have been experiencing a ranking drop every two months, so I came upon a new theory this morning... Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content? The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board. Thoughts? What would be the best way to block out your /dev area?
Web Design | | BoulderJoe0 -
Need advice on diplaying content for Search
Hi every body, I am doing landing page redesign(s). Does any body know or can refer a content carousel that can rotate video and pictures? The "site with images" search option result is a compelling reason to showcase pictures if your space competeiveness (showroom, merchandise, etc) can be improved with a strong image presence. here is my main landing page http://www.shearerpainting.com I know there is alot of stuff, and confusing call to action, but I am looking for strategies to clean it up, clear fous on action (get bid, learn more), but allow users to see that they can dig for more content.
Web Design | | johnshearer0
