Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Block parent folder in robot.txt, but not children
-
Example:
I want to block this URL (which shows up in Webmaster Tools as an error):
http://www.siteurl.com/news/events-calendar/usa
But not this:
-
The idea from Andrew is nice, but my guess would be that you're targeting multiple events so that might run into issues. What you could do is add some more regular expression and make it like this:
Disallow: ^/news/events-calendar/usa$
-
You could use "allow" in your robots.txt file for just this problem.
allow: news/events-calendar/usa/event-name
disallow: /news/events-calendar/usa
See the allow directive section of this page: https://en.wikipedia.org/wiki/Robots_exclusion_standard
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Has anyone ever tested to see if having an ads.txt file provided any SEO lift?
I know that the ads.txt system is designed to prevent ad fraud and technically has nothing to do with search. That said, the presence of such a file would seem to be an indicator of overall site quality because it would show that a site owner wants to participate in a fraud-free system. Has anyone ever tested that? If so, they don't seem to have published their results. Maybe it's a secret weapon that some pros are using and not sharing?
Web Design | | scodtt0 -
Have Your Thoughts Changed Regarding Canonical Tag Best Practice for Pagination? - Google Ignoring rel= Next/Prev Tagging
Hi there, We have a good-sized eCommerce client that is gearing up for a relaunch. At this point, the staging site follows the previous best practice for pagination (self-referencing canonical tags on each page; rel=next & prev tags referencing the last and next page within the category). Knowing that Google does not support rel=next/prev tags, does that change your thoughts for how to set up canonical tags within a paginated product category? We have some categories that have 500-600 products so creating and canonicalizing to a 'view all' page is not ideal for us. That leaves us with the following options (feel it is worth noting that we are leaving rel=next / prev tags in place): Leave canonical tags as-is, page 2 of the product category will have a canonical tag referencing ?page=2 URL Reference Page 1 of product category on all pages within the category series, page 2 of product category would have canonical tag referencing page 1 (/category/) - this is admittedly what I am leaning toward. Any and all thoughts are appreciated! If this were in relation to an existing website that is not experiencing indexing issues, I wouldn't worry about these. Given we are launching a new site, now is the time to make such a change. Thank you! Joe
Web Design | | Joe_Stoffel1 -
Reason for robots.txt file blocking products on category pages?
Hi I have a website with thosands of products. On the category pages, all the products are linked to with the code “?cgid” in the URL. But “?cgid” is also blocked in the robots.txt file for some reason. So I'm thinking it's stopping all my products getting crawled by Google. Am I right here? Is there any reason why a website would want to limit so many URL's? I'm only here a week and the sites getting great traffic, so don't want to go breaking it!!! Thanks
Web Design | | Frankie-BTDublin0 -
Why is my financial services site being flagged as gambling
Watchguard and Websense/Forecepoint are flagging my financial services site gambling...how can I prevent that from happening. https://fwag.com/
Web Design | | AdsposureDev0 -
Payment Options in Footer
Hi Moz Community. I'm currently working on cleaning up a website footer on an e-commerce site. I was curious to find out if including payment options is important? Either text or image icons. Including Visa, AMEX etc. Are there benefits of displaying them? Image attached. Thanks vcdoL
Web Design | | Kdruckenbrod0 -
Migration from HTML to Wordpress - SEO Implications?
I am in the process of having a wordpress site developed to replace my current HTML site. (I currently have my website in html and a blog in wordpress in a sub directory). I am doing this in phases to try and preserve as much of my good rankings as possible. My first phase is to replicate my site with the exact same pages, meta data, and site structure. I'm hoping that google will see this as not much change and not change my rankings for the worse. I also made it a goal that my site speed tests be at least equal to what they are now. We will have to 301 all of the URLs however since it will be going from /example.html to /example. I believe my blog will also need to move into the root directory as well, so I need to 301 all of those pages. I plan to wait a couple months for Phase 2. Phase 2 involves replacing old content (photo galleries), and introducing new content (virtual tours, videos, new pages, etc.) One of my reasons for moving to wordpress is to keep up with current trends a little easier since I have very little time. (I am owner, website maintainer, SEO - all on my own). My question here is three parts. 1. Do you think this strategy will work to preserve my current rankings? 2. Do you have any lessons learned or advice to share with me to make this as smooth as possible? 3. Do I really need to wait to add new content? I might get antsy and want to do it sooner! 🙂 Thank you in advance!
Web Design | | CalicoKitty20001 -
How to rewrite/redirect a folder name with .htaccess
I have a folder in my site that I want to rename. I don’t want to just rewrite the URL and keep my old folder name, I want to change the folder name and then do whatever is necessary with .hataccess to not lose search engine rankings. The folder name I want to change has a space in it and also is misspelled (whoops x2) Example. Mysite.com/old foldr/page.html Mysite.com/newfolder/page.html How would I go about doing this with .htaccess? Do I just switch the folder name on my server and then set up a redirect, or do I do a rewrite? Sorry now familiar with the terms or .htacces. Thanks all
Web Design | | SheffieldMarketing0 -
Should the parent directory of the main site-navigation be clickable or not?!?
Highly discussed in our team is the question: Should all parent navigation items be clickable, or only the ones that have no child menu appearing on mouse over? At Starwood Germany, we would like to adjust the main navigation for all our websites in order to improve consistency and user friendliness. At the moment, most of our websites feature both clickable non-clickable parent items, depending on whether the items have a corresponding child menu (appearing on mouse over) or not. See example here: http://www.imperialvienna.com/en Some of our team members believe it might be irritating and/or confusing for the user if some items are clickable while others are not. What do you think? Any thoughts and insights would be truly appreciated!
Web Design | | DFM_GSA0