Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Confirming Robots.txt code deep Directories

Technical SEO

536

cbielich last edited by

Just want to make sure I understand exactly what I am doing

If I place this in my Robots.txt

Disallow: /root/this/that

By doing this I want to make sure that I am ONLY blocking the directory /that/ and anything in front of that. I want to make sure that /root/this/ still stays in the index, its just the that directory I want gone.

Am I correct in understanding this?
1 Reply Last reply
Reply Quote 0
MoosaHemani Banned last edited by

that's right!

Disallow: /root/this/ will bock the complete directly whereas,

Disallow: /root/this/that will only block the "that" within "this"

hope this helps!
1 Reply Last reply
Reply Quote 2

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Blocking pages from Moz and Alexa robots

Hello, We want to block all pages in this directory from Moz and Alexa robots - /slabinventory/search/ Here is an example page - https://www.msisurfaces.com/slabinventory/search/granite/giallo-fiesta/los-angeles-slabs/msi/ Let me know if this is a valid disallow for what I'm trying to. User-agent: ia_archiver
Disallow: /slabinventory/search/* User-agent: rogerbot
Disallow: /slabinventory/search/* Thanks.
Technical SEO | | Pushm

0
Robots.txt Tester - syntax not understood

I've looked in the robots.txt Tester and I can see 3 warnings: There is a 'syntax not understood' warning for each of these. XML Sitemaps:
https://www.pkeducation.co.uk/post-sitemap.xml
https://www.pkeducation.co.uk/sitemap_index.xml How do I fix or reformat these to remove the warnings? Many thanks in advance.
Jim
Technical SEO | | JamesHancocks1

0
No index tag robots.txt

Hi Mozzers, A client's website has a lot of internal directories defined as /node/*. I already added the rule 'Disallow: /node/*' to the robots.txt file to prevents bots from crawling these pages. However, the pages are already indexed and appear in the search results. In an article of Deepcrawl, they say you can simply add the rule 'Noindex: /node/*' to the robots.txt file, but other sources claim the only way is to add a noindex directive in the meta robots tag of every page. Can someone tell me which is the best way to prevent these pages from getting indexed? Small note: there are more than 100 pages. Thanks!
Jens
Technical SEO | | WeAreDigital_BE

0
Google Indexing Development Site Despite Robots.txt Block

Hi, A development site that has been set-up has the following Robots.txt file: User-agent: * Disallow: / In an attempt to block Google indexing the site, however this isn't the case and the development site has since been indexed. Any clues why this is or what I could do to resolve it? Thanks!
Technical SEO | | CarlWint

0
Confirmation of penalty

Hi, I have been discussing with friends about possible penalties on few domains [listed below]. I would appreciate if any of seomoz staff could check them and explain if these sites are penalized in full, or part of these websites are penalized or homepage being penalized, any identification of penalty being imposed, and how to get these sites restored. http://www.google.com.pk/#sclient=psy-ab&q=123businesslistings.com&oq=123businesslistings.com&gs_l=hp.3...4797.4797.1.5047.1.1.0.0.0.0.0.0..0.0...0.0...1c.1.8.psy-ab.xDG3ytsYUVU&pbx=1&bav=on.2,or.r_qf.&bvm=bv.45107431,d.aWc&fp=db751f8c4669b6fe&biw=1441&bih=603 keyword searched=123businesslistings.com
Technical SEO | | ClaudioJudsoni

0
Use of Robots.txt file on a job site

We are performing SEO on a large niche Job Board. My question revolves around the thought of no following all the actual job postings from their clients as they only last for 30 to 60 days. Anybody have any idea on the best way to handle this?
Technical SEO | | WebTalent

0
How can I exclude display ads from robots.txt?

Google has stated that you can do this to get spiders to content only, and faster. Our IT guy is saying it's impossible.
Do you know how to exlude display ads from robots.txt? Any help would be much appreciated.
Technical SEO | | GregBeddor

0
Restricted by robots.txt and soft bounce issues (related).

In our web master tools we have 35K (ish) URLs that are restricted by robots.txt and as have 1200(ish) soft 404s. WE can't seem to figure out how to properly resolve these URLs so that they no longer show up this way. Our traffic from SEO has taken a major hit over the last 2 weeks because of this. Any help? Thanks, Libby
Technical SEO | | GristMarketing

0