How do I disallow crawl on a directory when it's a prefix to my site's URL?
-
I am trying to disallow our media repository (hosted elsewhere, but appears as a directory on our site) from being crawled by robots but it is not a subdirectory of the site, it's a prefix.
So I need to disallow: mediabank.mywebsite.org
Not: mysite.org/mediabank
What would I need to put in my robots.txt and/or the other host's robots.txt to make this happen?
Thanks!
-
Hey there! Tawny from Moz's Help Team here.
You'll want to add a robots.txt file for that subdomain, and then add a Disallow command to that robots.txt file. So, using your example, you'd want a file like mediabank.mywebsite.org/robots.txt that had a Disallow command for any robots you don't want crawling that subdomain.
For all user-agents, that would look something like this:
User-agent: *
Disallow: /That would stop any user-agents from crawling any pages on that subdomain.
I hope this helps! If you've still got questions, feel free to send us a note at help@moz.com and we'll do our best to sort things out for you.
-
Hi,
Please check this old thread on the same topic @ https://moz.com/community/q/block-an-entire-subdomain-with-robots-txt
Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is on-demand crawl reporting numbers that are very different than the regualr site crawl by Moz?
When I look at the site crawl available through my dashboard, it reports 22.9K pages crawled. I just ran a on-demand crawl and it is only reporting 6K pages crawled. Why is the number so much lower?
Moz Bar | | ctripp10100 -
Site search solution?
Hey guys, since google site search is going away we are looking for an alternative. I saw that you guys are using a pretty nice looking site search solution. Can you tell me what it is?
Moz Bar | | snapstream0 -
MozBar Issues? Can't get info even logged in
Anyone having issues with the Moz bar? Lately no matter how many times I log in, getting data is difficult as it keeps asking me to create an acct. or log in. Even logged into Moz and on Q&A it is asking me to log in. THanks
Moz Bar | | RobertFisher0 -
Does Moz's keyword tool pull data from your IP address?
Does anyone know how Moz's keyword tool pulls their keyword ranks? Do they take it based off of the IP (history and cookies) that is being used? I am trying to find a way to collect keyword data that is neutral and not based off of my previous searches, etc. TIA
Moz Bar | | ReviveMedia0 -
804 : HTTPS (SSL) Error in Crawl Test
So I am getting this 804 Error but I have checked our Security Certificate and it looks to be just fine. In fact we have another 156 days before renewal on it. We did have some issues with this a couple months ago but it has been fixed. Now, there is a 301 from http to https and I did not start the crawl on https so I am curious if that is the issue? Just wanted to know if anybody else has seen this and if you were able to remedy it? Thanks,
Moz Bar | | DRSearchEngOpt
Chris Birkholm0 -
How do I go about fixing my High Priority issues that SEO moz says I have on a PHP site?
I am been trying to deal with this problem for some time now. I have talked to several IT people and SEO moz. None seem to know how to fix these issues on the type of site our company is. Our biggest issue with is Duplicate Page Content. We also have some title issues. Our site is built with PHP coding and variable, meaning the site is not a typical static website. We have a handful of pages that are dynamic depending on what the users chooses to see and do. So, my problem is I can't just go to a specific page and put the canonical or the redirect. It isn't multiple pages for our category pages, for example, it is just one that builds the page depending on the search. Please help!
Moz Bar | | JoshMaxAmps0 -
Does anyone have a good article or video on how to read the SEO MOZ crawl report column by column?
I am trying to find a good how-to on how to read and analyze each column of the SEO MOZ crawl report, specifically, the excel sheet it allows you to export. What I'm really trying to get to the bottom of is what the "Yes" indiciates under rel-cononical. If it says "yes," does this mean that the link in question has been canonoicalized?
Moz Bar | | armcwill0 -
Understanding Onsite Elements and Moz's Onpage Grader
I'm currently helping a friend with his website. One of the pages I am targeting for his business is: http://stalbansdentist.com.au/dentist-cairnlea/ The main keyword this page is targeting will be "dentist cairnlea" (Cairnlea is a suburb near Melbourne, Australia) To perfect onsite optimization, I've used the Moz Onpage Grader. Generally, its straightforward, but I have encountered some issues... Appropriate Use of Rel Canonical The page does have the following tag: <link rel="<a class="attribute-value">canonical</a>" href="http://stalbansdentist.com.au/dentist-cairnlea/" />. I thought this would suffice. What exactly do I need to do to fix this critical problem? Appropriate Characters in the URL I don't understand this one. The URL has completely appropriate characters. Why does Moz Onpage Grader insist that URLs here need to be appropriated? No More Than One H1 Tag There appears to be only one H1 tag on this page. Is Moz just wrong on this one or perhaps a little delayed? There were 2 H1 tags before one was deleted and a recheck done with this problem remaining on Moz's onsite check. Any assistance here on these 3 points and just understanding the Moz Onpage grader would be appreciated!
Moz Bar | | Gavo0