Using 2 wildcards in the robots.txt file
-
I have a URL string which I don't want to be indexed. it includes the characters _Q1 ni the middle of the string.
So in the robots.txt can I use 2 wildcards in the string to take out all of the URLs with that in it? So something like /_Q1. Will that pickup and block every URL with those characters in the string?
Also, this is not directly of the root, but in a secondary directory, so .com/.../_Q1. So do I have to format the robots.txt as //_Q1* as it will be in the second folder or just using /_Q1 will pickup everything no matter what folder it is on?
Thanks.
-
I'm not 100% positive, however it does make sense to use it this way.
User-agent: *
Disallow: /*_Q1$
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots blocked by pages webmasters tools
a mistake made in software. How can I solve the problem quickly? help me. XTRjH
Intermediate & Advanced SEO | | mihoreis0 -
2 Title elements on a single page
Hi there. I am looking at a site that currently has 2 <title>elements present on every page through out the site. They have a main Title with appropriate optimization. Then they have another <title> that appears to be empty. It looks like this; <span class="html-tag"><title></span><span class="html-tag"></title> I know having 2 <title>elements is not ideal. But if the second one is empty, can it still have a negative affect? </p> <p>Thanks</p></title>
Intermediate & Advanced SEO | | Ciceron_main0 -
Linking Same Page using Important Keyword as Anchor?
I have came across sites that links to same page from a textual part using their topmost keywords. What is the benefit of linking same page using anchor as keywords to the same page? Does it give SEO benefits?
Intermediate & Advanced SEO | | welcomecure0 -
Should we use the rel-canonical tag?
We have a secure version of our site, as we often gather sensitive business information from our clients. Our https pages have been indexed as well as our http version. Could it still be a problem to have an http and an https version of our site indexed by Google? Is this seen as being a duplicate site? If so can this be resolved with a rel=canonical tag pointing to the http version? Thanks
Intermediate & Advanced SEO | | annieplaskett1 -
Stuck on Page 2 - What Would You Do?!?
My site is : http://goo.gl/JgK1e My main keyword is : Plastic Bins i have been going back and forth between page 1 and 2 for this keyword and i was wondering if any of you could provide any guidance as to why i can't get on the top of page 1, and stay there... My site has been around for a while, we believe we have a great user experience, all unique, fresh content, and the lowest prices... I must be missing out on something major if I cannot get a steady page 1 ranking... Any thoughts? Thanks in advance...
Intermediate & Advanced SEO | | Prime850 -
Do you use your own Blog networks?
Do you use a network of sites you own for links to your clients in your seo efforts? I see so many seo companies doing this from such junk sites with all their clients in the blog roll, it seems totally crazy. It seems this stuff works do any of you do this if so how do you keep it white hat?
Intermediate & Advanced SEO | | DavidKonigsberg0 -
Search Engine Blocked by robots.txt for Dynamic URLs
Today, I was checking crawl diagnostics for my website. I found warning for search engine blocked by robots.txt I have added following syntax to robots.txt file for all dynamic URLs. Disallow: /*?osCsid Disallow: /*?q= Disallow: /*?dir= Disallow: /*?p= Disallow: /*?limit= Disallow: /*review-form Dynamic URLs are as follow. http://www.vistastores.com/bar-stools?dir=desc&order=position http://www.vistastores.com/bathroom-lighting?p=2 and many more... So, Why should it shows me warning for this? Does it really matter or any other solution for these kind of dynamic URLs.
Intermediate & Advanced SEO | | CommercePundit0 -
HTML5 and using multiple H1 tags
Hi All, Our dev team have just asked me a very interesting question........ Within the context of an HTML5 page, where it is supported and encouraged to use multiple H1 tags, will the use of multiple H1 tags be detrimental to SEO? or does Google fully understand how HTML5 works and therefore not penalise a website for using multiple H1 tags? I have an opinion on this that if it helps usability and user experience then it is likely that it will be good for SEO. It would be really good to hear views of people who have tried this or have decided against it! Thanks
Intermediate & Advanced SEO | | A_Q0
