Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Block parent folder in robot.txt, but not children
-
Example:
I want to block this URL (which shows up in Webmaster Tools as an error):
http://www.siteurl.com/news/events-calendar/usa
But not this:
-
The idea from Andrew is nice, but my guess would be that you're targeting multiple events so that might run into issues. What you could do is add some more regular expression and make it like this:
Disallow: ^/news/events-calendar/usa$
-
You could use "allow" in your robots.txt file for just this problem.
allow: news/events-calendar/usa/event-name
disallow: /news/events-calendar/usa
See the allow directive section of this page: https://en.wikipedia.org/wiki/Robots_exclusion_standard
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Questions about putting 2-3 keywords in title tag.
Hi there, what is the best way to decide the keywords to be used in title tag?
Web Design | | lucybish0 -
Has anyone ever tested to see if having an ads.txt file provided any SEO lift?
I know that the ads.txt system is designed to prevent ad fraud and technically has nothing to do with search. That said, the presence of such a file would seem to be an indicator of overall site quality because it would show that a site owner wants to participate in a fraud-free system. Has anyone ever tested that? If so, they don't seem to have published their results. Maybe it's a secret weapon that some pros are using and not sharing?
Web Design | | scodtt0 -
Why is my financial services site being flagged as gambling
Watchguard and Websense/Forecepoint are flagging my financial services site gambling...how can I prevent that from happening. https://fwag.com/
Web Design | | AdsposureDev0 -
Payment Options in Footer
Hi Moz Community. I'm currently working on cleaning up a website footer on an e-commerce site. I was curious to find out if including payment options is important? Either text or image icons. Including Visa, AMEX etc. Are there benefits of displaying them? Image attached. Thanks vcdoL
Web Design | | Kdruckenbrod0 -
Migration from HTML to Wordpress - SEO Implications?
I am in the process of having a wordpress site developed to replace my current HTML site. (I currently have my website in html and a blog in wordpress in a sub directory). I am doing this in phases to try and preserve as much of my good rankings as possible. My first phase is to replicate my site with the exact same pages, meta data, and site structure. I'm hoping that google will see this as not much change and not change my rankings for the worse. I also made it a goal that my site speed tests be at least equal to what they are now. We will have to 301 all of the URLs however since it will be going from /example.html to /example. I believe my blog will also need to move into the root directory as well, so I need to 301 all of those pages. I plan to wait a couple months for Phase 2. Phase 2 involves replacing old content (photo galleries), and introducing new content (virtual tours, videos, new pages, etc.) One of my reasons for moving to wordpress is to keep up with current trends a little easier since I have very little time. (I am owner, website maintainer, SEO - all on my own). My question here is three parts. 1. Do you think this strategy will work to preserve my current rankings? 2. Do you have any lessons learned or advice to share with me to make this as smooth as possible? 3. Do I really need to wait to add new content? I might get antsy and want to do it sooner! 🙂 Thank you in advance!
Web Design | | CalicoKitty20001 -
Bing Indexation and handling of X-ROBOTS tag or AngularJS
Hi MozCommunity, I have been tearing my hair out trying to figure out why BING wont index a test site we're running. We're in the midst of upgrading one of our sites from archaic technology and infrastructure to a fully responsive version.
Web Design | | AU-SEO
This new site is a fully AngularJS driven site. There's currently over 2 million pages and as we're developing the new site in the backend, we would like to test out the tech with Google and Bing. We're looking at a pre-render option to be able to create static HTML snapshots of the pages that we care about the most and will be available on the sitemap.xml.gz However, with 3 completely static HTML control pages established, where we had a page with no robots metatag on the page, one with the robots NOINDEX metatag in the head section and one with a dynamic header (X-ROBOTS meta) on a third page with the NOINDEX directive as well. We expected the one without the meta tag to at least get indexed along with the homepage of the test site. In addition to those 3 control pages, we had 3 pages where we had an internal search results page with the dynamic NOINDEX header. A listing page with no such header and the homepage with no such header. With Google, the correct indexation occured with only 3 pages being indexed, being the homepage, the listing page and the control page without the metatag. However, with BING, there's nothing. No page indexed at all. Not even the flat static HTML page without any robots directive. I have a valid sitemap.xml file and a robots.txt directive open to all engines across all pages yet, nothing. I used the fetch as Bingbot tool, the SEO analyzer Tool and the Preview Page Tool within Bing Webmaster Tools, and they all show a preview of the requested pages. Including the ones with the dynamic header asking it not to index those pages. I'm stumped. I don't know what to do next to understand if BING can accurately process dynamic headers or AngularJS content. Upon checking BWT, there's definitely been crawl activity since it marked against the XML sitemap as successful and put a 4 next to the number of crawled pages. Still no result when running a site: command though. Google responded perfectly and understood exactly which pages to index and crawl. Anyone else used dynamic headers or AngularJS that might be able to chime in perhaps with running similar tests? Thanks in advance for your assistance....0 -
How to rewrite/redirect a folder name with .htaccess
I have a folder in my site that I want to rename. I don’t want to just rewrite the URL and keep my old folder name, I want to change the folder name and then do whatever is necessary with .hataccess to not lose search engine rankings. The folder name I want to change has a space in it and also is misspelled (whoops x2) Example. Mysite.com/old foldr/page.html Mysite.com/newfolder/page.html How would I go about doing this with .htaccess? Do I just switch the folder name on my server and then set up a redirect, or do I do a rewrite? Sorry now familiar with the terms or .htacces. Thanks all
Web Design | | SheffieldMarketing0 -
IP block in Google
Our office has a number of people performing analysis and research on keyword positions, volume, competition etc. We have 1 external static IP address. We installed the static IP so we can filter out our visits in Google Analytics. However by 10 AM we get impssible CAPTCHA's or even get blocked in Google. Do you have any experience with such an issue? Any solutions you can recommend? Any help would be appreciated! SXI5A.png
Web Design | | Partouter0