Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How to allow one directory in robots.txt

Technical SEO

835

poolguy last edited by

Hello, is there a way to allow a certain child directory in robots.txt but keep all others blocked?

For instance, we've got external links pointing to /user/password/, but we're blocking everything under /user/. And there are too many /user/somethings/ to just block every one BUT /user/password/.

I hope that makes sense...

Thanks!
1 Reply Last reply
Reply Quote 0
EricaMcGillivray last edited by

Yes, you can set it up like this:

Disallow: /user/ Allow: /user/password/

And that should do it!
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

What Metadata should one use multi country directory

Currently this is what applies throughout the site. property="og:locale" content="en_GB" /> How would one set this for properties in Italy or Spain for example? (The language is all in English) Regards Tai
Technical SEO | | Taiger

0
Switchboard Tags - Multiple desktop pages pointing to one mobile page

I have recently started to implement switchboard tags to connect our mobile and desktop pages, and to ensure that our mobile pages show up in rankings for mobile users. Because our desktop site is much deeper in content than our mobile site, there are a number of desktop pages we would like to have point to one mobile page. However, with the switchboard tags, this poses a problem because it requires multiple rel=canonical tags to be placed on the one mobile page. I'm assuming this will either confuse the search engines, or they will choose to ignore the rel=canonical tag altogether. Any ideas on how to approach this situation other than creating an equivalent mobile version of every desktop page or implementing a user agent detection redirect?
Technical SEO | | JBlank

0
Robots.txt

www.mywebsite.com**/details/**home-to-mome-4596 www.mywebsite.com**/details/**home-moving-4599 www.mywebsite.com**/details/**1-bedroom-apartment-4601 www.mywebsite.com**/details/**4-bedroom-apartment-4612 We have so many pages like this, we do not want to Google crawl this pages So we added the following code to Robots.txt User-agent: Googlebot Disallow: /details/ This code is correct?
Technical SEO | | iskq

0
My whole directory dropped from google

Hi there I have lost a whole directory in Google rankings and this is where my traffic was coming from. Do you know why this would happen? It is the eCommerce part of the site. So if I do site:www.domain.com the site is there and then site:www.domain.com/directory is not there any more. Thanks for your help.
Technical SEO | | vcasebourne

0
Robots.txt checker

Google seems to have discontinued their robots.txt checker. Is there another tool that I can use to check my text instead? Thanks!
Technical SEO | | theLotter

0
How many times robots.txt gets visited by crawlers, especially Google?

Hi, Do you know if there's any way to track how often robots.txt file has been crawled? I know we can check when is the latest downloaded from webmaster tool, but I actually want to know if they download every time crawlers visit any page on the site (e.g. hundreds of thousands of times every day), or less. thanks...
Technical SEO | | linklater

0
Hiding sub directory from a url string

I have been trying for a few hours now to hide a sub-directory from a url string on a joomla site using .htaccess every method i have found either has no effect or breaks the site. i am aiming to change all links with component in them: http://mysite.com/component/music/7335 to http://mysite.com/music/7335 I am using the latest Joomla 2.5 install but not sure how or if that affects anything. Any help would be greatly appreciated
Technical SEO | | omega

0
Am I missing something if I absorb one site into another?

We are absorbing one of our sites into another and I want to make sure I am not missing anything. The site that is being absorbed will no longer exist as all the content has been replicated/duplicated on the main site. About a month ago we added canonicals to all the duplicate content to point to the new site it will be a part of. That went very well and organic traffic continued to flow to those pages on the new site. We recently (yesterday) used 301s on all the pages using mod_rewrites and redirected the domain name from the old site name to the new one. Using mod_rewrites we redirect any other page linking to that domain to www.newsite.com?ref=oldsite.com. So far I dont see anything coming in on an unexpected link. I still need to tell Google via Webmaster Tools that the oldsite is now on newsite.com correct? Is there anything else I might bump into that we havent thought of? Thanks
Technical SEO | | GeorgeLaRochelle

0