Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Block parent folder in robot.txt, but not children
-
Example:
I want to block this URL (which shows up in Webmaster Tools as an error):
http://www.siteurl.com/news/events-calendar/usa
But not this:
-
The idea from Andrew is nice, but my guess would be that you're targeting multiple events so that might run into issues. What you could do is add some more regular expression and make it like this:
Disallow: ^/news/events-calendar/usa$
-
You could use "allow" in your robots.txt file for just this problem.
allow: news/events-calendar/usa/event-name
disallow: /news/events-calendar/usa
See the allow directive section of this page: https://en.wikipedia.org/wiki/Robots_exclusion_standard
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Questions about putting 2-3 keywords in title tag.
Hi there, what is the best way to decide the keywords to be used in title tag?
Web Design | | lucybish0 -
Does changing content and design of the website gonna affect my all the backlinks i have made till now
i have been working on my link profile for a month now, after learning about 5 step moz methodology i have decided that i would like to change all of the content of my site and taylor it to what my customers need, am i gonna loose all the domain authority if make changes? if it gonna affect, hows that gonna come out
Web Design | | calvinkj0 -
Has anyone ever tested to see if having an ads.txt file provided any SEO lift?
I know that the ads.txt system is designed to prevent ad fraud and technically has nothing to do with search. That said, the presence of such a file would seem to be an indicator of overall site quality because it would show that a site owner wants to participate in a fraud-free system. Has anyone ever tested that? If so, they don't seem to have published their results. Maybe it's a secret weapon that some pros are using and not sharing?
Web Design | | scodtt0 -
Reason for robots.txt file blocking products on category pages?
Hi I have a website with thosands of products. On the category pages, all the products are linked to with the code “?cgid” in the URL. But “?cgid” is also blocked in the robots.txt file for some reason. So I'm thinking it's stopping all my products getting crawled by Google. Am I right here? Is there any reason why a website would want to limit so many URL's? I'm only here a week and the sites getting great traffic, so don't want to go breaking it!!! Thanks
Web Design | | Frankie-BTDublin0 -
What’s the best tool to visualize internal link structure and relationships between pages on a single site?
I‘d like to review the internal linking structure on my site. Is there a tool that can visualize the relationships between all of the pages within my site?
Web Design | | QBSEO0 -
Migration from HTML to Wordpress - SEO Implications?
I am in the process of having a wordpress site developed to replace my current HTML site. (I currently have my website in html and a blog in wordpress in a sub directory). I am doing this in phases to try and preserve as much of my good rankings as possible. My first phase is to replicate my site with the exact same pages, meta data, and site structure. I'm hoping that google will see this as not much change and not change my rankings for the worse. I also made it a goal that my site speed tests be at least equal to what they are now. We will have to 301 all of the URLs however since it will be going from /example.html to /example. I believe my blog will also need to move into the root directory as well, so I need to 301 all of those pages. I plan to wait a couple months for Phase 2. Phase 2 involves replacing old content (photo galleries), and introducing new content (virtual tours, videos, new pages, etc.) One of my reasons for moving to wordpress is to keep up with current trends a little easier since I have very little time. (I am owner, website maintainer, SEO - all on my own). My question here is three parts. 1. Do you think this strategy will work to preserve my current rankings? 2. Do you have any lessons learned or advice to share with me to make this as smooth as possible? 3. Do I really need to wait to add new content? I might get antsy and want to do it sooner! 🙂 Thank you in advance!
Web Design | | CalicoKitty20001 -
How to rewrite/redirect a folder name with .htaccess
I have a folder in my site that I want to rename. I don’t want to just rewrite the URL and keep my old folder name, I want to change the folder name and then do whatever is necessary with .hataccess to not lose search engine rankings. The folder name I want to change has a space in it and also is misspelled (whoops x2) Example. Mysite.com/old foldr/page.html Mysite.com/newfolder/page.html How would I go about doing this with .htaccess? Do I just switch the folder name on my server and then set up a redirect, or do I do a rewrite? Sorry now familiar with the terms or .htacces. Thanks all
Web Design | | SheffieldMarketing0 -
IP block in Google
Our office has a number of people performing analysis and research on keyword positions, volume, competition etc. We have 1 external static IP address. We installed the static IP so we can filter out our visits in Google Analytics. However by 10 AM we get impssible CAPTCHA's or even get blocked in Google. Do you have any experience with such an issue? Any solutions you can recommend? Any help would be appreciated! SXI5A.png
Web Design | | Partouter0