Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Confirming Robots.txt code deep Directories

Technical SEO

541

cbielich last edited by

Just want to make sure I understand exactly what I am doing

If I place this in my Robots.txt

Disallow: /root/this/that

By doing this I want to make sure that I am ONLY blocking the directory /that/ and anything in front of that. I want to make sure that /root/this/ still stays in the index, its just the that directory I want gone.

Am I correct in understanding this?
1 Reply Last reply
Reply Quote 0
MoosaHemani Banned last edited by

that's right!

Disallow: /root/this/ will bock the complete directly whereas,

Disallow: /root/this/that will only block the "that" within "this"

hope this helps!
1 Reply Last reply
Reply Quote 2

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Pages being flagged in Search Console as having a "no-index" tag, do not have a meta robots tag??

Hi, I am running a technical audit on a site which is causing me a few issues. The site is small and awkwardly built using lots of JS, animations and dynamic URL extensions (bit of a nightmare). I can see that it has only 5 pages being indexed in Google despite having over 25 pages submitted to Google via the sitemap in Search Console. The beta Search Console is telling me that there are 23 Urls marked with a 'noindex' tag, however when i go to view the page source and check the code of these pages, there are no meta robots tags at all - I have also checked the robots.txt file. Also, both Screaming Frog and Deep Crawl tools are failing to pick up these urls so i am a bit of a loss about how to find out whats going on. Inevitably i believe the creative agency who built the site had no idea about general website best practice, and that the dynamic url extensions may have something to do with the no-indexing. Any advice on this would be really appreciated. Are there any other ways of no-indexing pages which the dev / creative team might have implemented by accident? - What am i missing here? Thanks,
Technical SEO | | NickG-123

0
Is sitemap required on my robots.txt?

Hi, I know that linking your sitemap from your robots.txt file is a good practice. Ok, but... may I just send my sitemap to search console and forget about adding ti to my robots.txt? That's my situation: 1 multilang platform which means... ... 2 set of pages. One for each lang, of course But my CMS (magento) only allows me to have 1 robots.txt file So, again: may I have a robots.txt file woth no sitemap AND not suffering any potential SEO loss? Thanks in advance, Juan Vicente Mañanas Abad
Technical SEO | | Webicultors

0
Screaming Frog showing 503 status code. Why?

Screaming Frog is showing a 503 code for images. If I go and use a header checker like SEOBook it shows 200. Why would that be? Here is an example link- http://germanhausbarn.com/wp-content/uploads/2014/07/36-UPC-5145536-John-Deere-Stoneware-Logo-Mug-pair-25.00-Heavy-4-mugs-470x483.jpg
Technical SEO | | EcommerceSite

0
Why is robots.txt blocking URL's in sitemap?

Hi Folks, Any ideas why Google Webmaster Tools is indicating that my robots.txt is blocking URL's linked in my sitemap.xml, when in fact it isn't? I have checked the current robots.txt declarations and they are fine and I've also tested it in the 'robots.txt Tester' tool, which indicates for the URL's it's suggesting are blocked in the sitemap, in fact work fine. Is this a temporary issue that will be resolved over a few days or should I be concerned. I have recently removed the declaration from the robots.txt that would have been blocking them and then uploaded a new updated sitemap.xml. I'm assuming this issue is due to some sort of crossover. Thanks Gaz
Technical SEO | | PurpleGriffon

0
Search Term not appearing in On-Page Grader but is in code?

Hi, I have never had this issue in the past, but I am working on a website currently which is - http://hrfoundations.co.uk I didn't develop this website and the pages are in separate folders on the FTP which I have never come across before but hey ho. Anyway I have added the search term (HR Consultancy West Midlands) to the about us page in the h1, title and description however when I run the On-Page Grader it only says that I have entered it within the body which I haven't even done yet. When I view the page source it has defiantly been uploaded and the search terms are there, I have tried checking the spelling incase I spelt it incorrectly and I still can't seem to find the issue. Has anyone else experience this in the past, I'm going to go away now and check the code for any unclosed divs etc to see if that may be causing any issue's. Thanks Chris
Technical SEO | | chrissmithps

0
What directory should a site go in (url structure)?

Hi All, The is the first actual SEO campaign i've worked on and I had a few question about where the site should live on the server and url structure. The site is in WP and we're using Yoast SEO. Anyway the site lives in a a folder called Coastal, which is a child of the WWW folder. So the permalink of the homepage is mcoastalwindows.com/coastal/. The URL is mycoastalwindows.com. The thing is I can still get to the homepage or any of the pages on the site by typing in the /coastal/. Another example is permalink mycoastalwndows.com/coastal/siding/ and url mycoastalwindows.com/siding/. The urls always display without the /coastal/, so I'm not too worried about people linking to them, but Yoast puts a canonical element to the permalink and always includes the /coastal/. Also I'm seeing that Google displays a lot of the urls with the /coastal/, which is an issue seeing as we don't link to the pages that way. My original thought was to solve this at the source and just move everything out of the coastal directory, but the developer swears that it's more secure being in another folder especially with WP. What would you all do and what is best practice? Would you move everything out of the coastal folder, 301 re-direct, do something with. htaccess, or another solution? Appreciate the input thanks!
Technical SEO | | Mario.Souza

0
Doubt between sub-directory and sub-domain for develop Blog for my business website

Hi, I am much aware with SEO. But still there is a confusion in my mind. I am in the process of making blog for my Business website. I am confused between whether my blog should be with sub-directory(www.metaoption.com/ blog) or with sub-domain(blog. metaoption.com). There are many articles on this topic but they have created much confusions. Please suggest me the best option and also give me reason behind it if you can. Thanks
Technical SEO | | Perfect007

0
Schema coding

Hi, I was wondering if you may know if you have to keep to the and coding when adding schema code to the site. For example if I'm already using H and P tags can I add the "itemprop" to those or do they have to be in aor as in the example below: <span itemprop="name">Kenmore White 17" Microwavespan>
Product description:
<span itemprop="description">0.7 cubic feet countertop microwave. Has six preset cooking categories and convenience features like Add-A-Minute and Child Lock.span> So could I code it like this? <h1 itemprop="name">Kenmore White 17" Microwaveh1>
Product description:
<p itemprop="description">0.7 cubic feet countertop microwave. Has six preset cooking categories and convenience features like Add-A-Minute and Child Lock.p> Thank you,
Etela
Technical SEO | | DragonSearch

0