Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do we need to interlink homepage to the highest?
Hi, So our homepage is not even at top 5 pages interlinked most. I could see a ranking drop even when other pages overtaken homepage in internal linking. Homepage is the highest priority page we are expected to rank for our primary keyword. So, not linking homepage internally high but giving importance to other pages confuses Google and outrank homepage in rankings? Thanks
Web Design | | vtmoz0 -
Is this spammy/panda problem?
We have a property site and have many area pages with listings of properties. For example, one page may have up to 30 properties listed, with 100 words of description on that listing page for that property and then you click 'more info' to get to the page of that property with maybe 200 words in total. I want to add bullet points for each property on the area page but Im worried that it may be seen by google as spammy even though its usefull to the client. For example, if I had 30 properties on a page, and 28 of them said next to each picture.. Property Type: Shared, Open Plan, Single Bed Would that be a problem for google?
Web Design | | nick-name1230 -
SEO strategy for UK / US websites
Hi, We currently have a UK-focused site on www.palmatin.com ; We're now targeting the North American market as well, but the contents of the site need to be different from UK. One option was to create another domain for the NA market but I assume it would be easier to rank with palmatin.com though. What would you suggest to do, if a company is targeting two different countries in the same language? thanks, jaan
Web Design | | JaanMSonberg0 -
Google pagespeed / lazy image load
Hi, we are using the apache module of google pagespeed. It works really great, helps a lot. But today I've asked me one question: Does the "lazy load" feature for images harm the ranking? The module reworks the page to load the images only if the are visible at the screen. Is this behavior also triggered by the google bot? Or are the images invisible for google? Any expirience about that? Best wishes, Georg.
Web Design | | GeorgFranz0 -
Wordpress themes causing google penalty(need experts to settle a debate)
Hi, I have been having a disagreement with another online marketing company. We are both promoting the same product under a different brand name but we ended up using the same theme to build our WordPress sites off of but in no way is the content the same. They are telling me that using the same theme in the same industry will cause a Google penalty. I do not believe this and do not see this causing a problem. The sites are relatively new so there is no proof of traffic dropping or penalties as of yet. What is everyone's professional opinion on this? Can a WordPress theme cause duplicate content penalty? If so would that not mean that anyone using themes will have some sort of penalty?
Web Design | | impact891 -
Can you use a base element and mod_rewrite to alleviate the need for absolute URLs?
This is a follow up question to Scott Parsons' question about using absolute versus relative URLs when linking internally. Andy King makes the statement that this can be done and that it saves additional space (which he claims then can improve page speed). Is this a true and accurate statement? Can using a base element and mod-rewrite alleviate the need for absolute URLs? I need to know before going off on a "change all of our relative URLs to absolutes" campaign. Thanks in advance! Dana
Web Design | | danatanseo0 -
How to verify http://bizdetox.com for google webmaster tools
Hey guys i tried to to make a Preferred Domain choice in webmaster tools, but it is not allowing me to save my choice bec its asking me to verify that i own http://bizdetox.com How do i go about doing that and what are the steps I have already verified www.bizdetox.com
Web Design | | BizDetox0 -
Critique a site I'm working on please
Hi guys, I have a list of things to do on this website that is longer than my arm but thought I would gather a few more thoughts on the site before moving on further with it. The site is here. It went live a couple of days ago for live testing. I have a news section coming very soon at theworkplacedepot.co.uk/news - it is in live testing now. One glaring issue for me is that some of the products simply aren't "buyable". What I mean by that is that the options are too vague and too vast - as an outsider I wouldnt have a clue what I was buying! Other issues: The banner at the bottom of the HP is a test - its design will be changing. The sitemap has some issues and will be addressed. There is no area yet to "log-in". However, critique for me pretty please! 🙂
Web Design | | MattJanaway0