Block parent folder in robot.txt, but not children
-
Example:
I want to block this URL (which shows up in Webmaster Tools as an error):
http://www.siteurl.com/news/events-calendar/usa
But not this:
-
The idea from Andrew is nice, but my guess would be that you're targeting multiple events so that might run into issues. What you could do is add some more regular expression and make it like this:
Disallow: ^/news/events-calendar/usa$
-
You could use "allow" in your robots.txt file for just this problem.
allow: news/events-calendar/usa/event-name
disallow: /news/events-calendar/usa
See the allow directive section of this page: https://en.wikipedia.org/wiki/Robots_exclusion_standard
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Has anyone ever tested to see if having an ads.txt file provided any SEO lift?
I know that the ads.txt system is designed to prevent ad fraud and technically has nothing to do with search. That said, the presence of such a file would seem to be an indicator of overall site quality because it would show that a site owner wants to participate in a fraud-free system. Has anyone ever tested that? If so, they don't seem to have published their results. Maybe it's a secret weapon that some pros are using and not sharing?
Web Design | | scodtt0 -
Will There Be Much Impact When Moving Site To New Root Folder?
Hi, ok so I have a pretty big site that is located on my sever /root/current-folder/. I want to rebuild the site completely as it's using software that is out of date and not our main focus anymore (OpenCart). We want to move to a Wordpress platform, but want to have as little impact on the SEO as possible. Our current strategy is: List all URLs/Titles/Meta indexed with Google on current site Create new folder on the server /root/new-folder/ My question is... if I move to a new folder on the server (same TLD) and then re-route the TLD to go to this new folder, will there be more of an impact on SEO that if I start a fresh in the current folder? Thanks
Web Design | | Easigrass0 -
Wordpress Theme is blocking alt tags. Does anybody know of any special plugins?
We have a special wordpress theme for nataliecass.com. Unfortunately the theme is blocking all the alt tags (this is a photography website...alt tags are very important). Does anybody know of any special WP plugins for alt tags? Thanks
Web Design | | VanguardCommunications0 -
Google tag manager on blocked beta site - will it phone home to Google and cause site to get indexed?
We want to develop a beta site, in a directory with the robots.txt blocking bots. We want to include the Google Tag Manager tags and event layer tracking code on this beta site. My question is that by including the Google Tag Manager code, that phones home to Google, will it cause Google to index this beta site when we don't want it indexed?
Web Design | | CFSSEO0 -
For A Corporation With 3 Distinct Business Divisions, Is It Better To Go With 1 Domain & 3 Sub-Domains, 1 Domain & 3 Folders, or 3 Domains for SEO Purposes?
Hi, I am working on a project right now for an existing client, we have one domain up and running well, they want to create an 'umbrella' site to cover three current business divisions and roll everything up under that main site, including the existing site on a totally different domain (would migrate over and 301 redirect from current domain). From what I've researched, I am inclined towards one main domain with three sub-domains due to the amount of content for each business division being significantly different enough that it seems to deserve separation from each other. However, in terms of SEO and maintaining consistent domain authority, would anyone recommend it be better to structure this as just folders/categories falling under the main domain instead of separate sub-domains for each division, and focus keyword targeting on pages tailored to that end within the main domain structure rather than spreading out link-juice to different sub-domains? Thanks!
Web Design | | Dan_InboundHorizons0 -
Robots.txt - Allow and Disallow. Can they be the same?
Hi All, I need some help on the following: Are the following commands the same? User-agent: * Disallow: or User-agent: * Allow: / I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots. Is that correct? Many thanks, Aidan
Web Design | | Presenter0 -
Should the parent directory of the main site-navigation be clickable or not?!?
Highly discussed in our team is the question: Should all parent navigation items be clickable, or only the ones that have no child menu appearing on mouse over? At Starwood Germany, we would like to adjust the main navigation for all our websites in order to improve consistency and user friendliness. At the moment, most of our websites feature both clickable non-clickable parent items, depending on whether the items have a corresponding child menu (appearing on mouse over) or not. See example here: http://www.imperialvienna.com/en Some of our team members believe it might be irritating and/or confusing for the user if some items are clickable while others are not. What do you think? Any thoughts and insights would be truly appreciated!
Web Design | | DFM_GSA0 -
Should /dev folder be blocked?
I have been experiencing a ranking drop every two months, so I came upon a new theory this morning... Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content? The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board. Thoughts? What would be the best way to block out your /dev area?
Web Design | | BoulderJoe0