Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How to allow one directory in robots.txt

Technical SEO

829

poolguy last edited by

Hello, is there a way to allow a certain child directory in robots.txt but keep all others blocked?

For instance, we've got external links pointing to /user/password/, but we're blocking everything under /user/. And there are too many /user/somethings/ to just block every one BUT /user/password/.

I hope that makes sense...

Thanks!
1 Reply Last reply
Reply Quote 0
EricaMcGillivray last edited by

Yes, you can set it up like this:

Disallow: /user/ Allow: /user/password/

And that should do it!
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Why did our highly ranked keyword drop to 51+ for just one week?

One of our most important keywords (ranked 5) dropped to 51+ one week and then went back to 5 around the time we launched a new site. Why did that happen?
Technical SEO | | virtuance_photography

0
Case study re-directing one site into another?

Hi there, We have access to a third party site that has high domain authority and we want to know if anyone has a case study demonstrating what happens when you 301-redirect a high DA site to another high DA site. In particular, we are wondering what kind of lift the site saw from the additional link equity?
Technical SEO | | nicole.healthline

0
Will a robots.txt disallow apply to a 301ed URL?

Hi there, I have a robots.txt query which I haven't tried before and as we're nearing a big time for sales, I'm hesitant to just roll out to live! Say for example, in my robots.txt I disallow the URL 'example1.html'. In reality, 'example1.html' 301s/302s to 'example2.html'. Would the robots.txt directive also apply to 'example2.html' (disallow) or as it's a separate URL, would the directive be ignored as it's not valid? I have a feeling that as it's a separate URL, the robots disallow directive won't apply. However, just thought I'd sense-check with the community.
Technical SEO | | ecommercebc

0
Why Google ranks a page with Meta Robots: NO INDEX, NO FOLLOW?

Hi guys, I was playing with the new OSE when I found out a weird thing: if you Google "performing arts school london" you will see w w w . mountview . org. uk at the 3rd position. The point is that page has "Meta Robots: NO INDEX, NO FOLLOW", why Google indexed it? Here you can see the robots.txt allows Google to index the URL but not the content, in article they also say the meta robots tag will properly avoid Google from indexing the URL either. Apparently, in my case that page is the only one has the tag "NO INDEX, NO FOLLOW", but it's the home page. so I said to myself: OK, perhaps they have just changed that tag therefore Google needs time to re-crawl that page and de-index following the no index tag. How long do you think it will take to don't see that page indexed? Do you think it will effect the whole website, as I suppose if you have that tag on your home page (the root domain) you will lose a lot of links' juice - it's totally unnatural a backlinks profile without links to a root domain? Cheers, Pierpaolo
Technical SEO | | madcow78

0
One-Pager and SEO

We're building a page that is going to feature over 31 people as difference makers in their field. We're unveiling one a day for an entire month. The very early mockup of the page has name, pic, some bio info, and a link to open up a new window with the full bio. I would love to have all of the bio content for all of the people on the page (and indexable), but I'm not sure how to do that while still being able to hide the full bios until they are expanded. Anybody have any tips that are SEO-friendly and/or examples of a page that is built like this and ranks well. Thanks!
Technical SEO | | spackle

0
Different IP's in one Server

Hi, I just want to ask if there is no bad effect in SEO if we do have different websites that has different IP address but has shared in only 1 server? Thank you
Technical SEO | | TirewebMarketing

0
How do I 301 redirect a number of pages to one page

I want to redirect all pages in /folder_A /folder_B to /folder_A/index.php. Can I just write one or two lines of code to .htaccess to do that?
Technical SEO | | Heydarian

0
Wildly different topics on one domain? (hobby blogger)

Hello Mozzers, Let's say I have a lot of knowledge about a range of different subjects, for example: bicycle maintenance wildlife photography tennis racket re-stringing brewing beer windsurfing holidays I would like to share all of my knowledge in blog format with lots of good content, instructions, videos etc. What is the best way to set up my blogging empire?!... a) Use separate domains for each subject: www.BRANDbicyclemaintenance.com, www.BRANDbrewingbeer.com b) Use one brand domain with sub directories: www.BRAND.com/bicycle-maintenance/, www.BRAND.com/brewing-beer/ b) Use sub-domains: bicyclemaintenance.BRAND.com, brewingbeer.BRAND.com (I would like to achieve good rankings & traffic in order to generate a small income from these sites)
Technical SEO | | Tman3

0