Using 2 wildcards in the robots.txt file
-
I have a URL string which I don't want to be indexed. it includes the characters _Q1 ni the middle of the string.
So in the robots.txt can I use 2 wildcards in the string to take out all of the URLs with that in it? So something like /_Q1. Will that pickup and block every URL with those characters in the string?
Also, this is not directly of the root, but in a secondary directory, so .com/.../_Q1. So do I have to format the robots.txt as //_Q1* as it will be in the second folder or just using /_Q1 will pickup everything no matter what folder it is on?
Thanks.
-
I'm not 100% positive, however it does make sense to use it this way.
User-agent: *
Disallow: /*_Q1$
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
2 pages competing
Hi all, My website currently has 2 pages that address the theme 'property investment in manchester' or 'buy to let manchester'. https://www.knightknox.com/developments/manchester/ https://www.knightknox.com/investments/manchester I am a bit concerned that we are competing against ourselves for these keywords. In my opinion the /investments page provides better content, but the /developments page ranks higher in Google. What do you think would be the best course of action?: Leave as is Merge the contents of both pages Redirect the /developments to /investments Or something else? Any ideas welcome. Thanks
Intermediate & Advanced SEO | | brian-madden0 -
Has anyone used Wildshark?
Just stumbled across a company called Wildshark SEO (http://www.wildshark.co.uk). Has anyone used them? Any feedback? Please and thank you!
Intermediate & Advanced SEO | | seoman100 -
Robots.txt
Hi all, Happy New Year! I want to block certain pages on our site as they are being flagged (according to my Moz Crawl Report) as duplicate content when in fact that isn't strictly true, it is more to do with the problems faced when using a CMS system... Here are some examples of the pages I want to block and underneath will be what I believe to be the correct robots.txt entry... http://www.XYZ.com/forum/index.php?app=core&module=search&do=viewNewContent&search_app=members&search_app_filters[forums][searchInKey]=&period=today&userMode=&followedItemsOnly= Disallow: /forum/index.php?app=core&module=search http://www.XYZ.com/forum/index.php?app=core&module=reports&rcom=gallery&imageId=980&ctyp=image Disallow: /forum/index.php?app=core&module=reports http://www.XYZ.com/forum/index.php?app=forums&module=post§ion=post&do=reply_post&f=146&t=741&qpid=13308 Disallow: /forum/index.php?app=forums&module=post http://www.XYZ.com/forum/gallery/sizes/182-promenade/small/ http://www.XYZ.com/forum/gallery/sizes/182-promenade/large/ Disallow: /forum/gallery/sizes/ Any help \ advice would be much appreciated. Many thanks Andy
Intermediate & Advanced SEO | | TomKing0 -
Is Tumblr being used as a social algorithm for Bing and Google?
Hi Y'all, Bing uses Pinterest as part of its image search by using pins of users...But what about Tumblr? Is Tubmlr still a good source for a ranking factor when pointing an image link to your blog? Our industry is pretty boring, so we would get a few clicks here and there a month, but nothing compared to Twitter, Facebook, Pinterest...but overall is Tumblr a good backlink every time a blog is posted for search algorithms?
Intermediate & Advanced SEO | | Shawn1240 -
What is the recommended way to save Image Files in WP?
Hi, Is there a recommended setting (or even plugin) to use when saving image files on my Wordpress blog?
Intermediate & Advanced SEO | | BeytzNet
Which folders etc. Thanks0 -
2 URLS pointing to the same content
Hi, We currently have 2 URL's pointing to the same website (long story why we have it) - A & B. A is our main website but we set up B as a rewrite URL to use for our Pay Per Click campaign. Now because its the same site, but B is just a URL rewrite, Google Webmaster Tools is seeing that we have thousands of links coming in from site B to site A. I want to tell Google to ignore site B url but worried it might affect site A. I can't add a no follow link on site B as its the same content so will also be applicable on Site A. I'm also worried about using Google Disavow as it might impact on site A! Can anyone make any suggestions on what to do, as I would like to hear from anyone with experience with this or can recommend a safe option. Thanks for your time!
Intermediate & Advanced SEO | | Party_Experts0 -
Meta canonical or simply robots.txt other domain names with same content?
Hi, I'm working with a new client who has a main product website. This client has representatives who also sells the same products but all those reps have a copy of the same website on another domain name. The best thing would probably be to shut down the other (same) websites and redirect 301 them to the main, but that's impossible in the minding of the client. First choice : Implement a conical meta for all the URL on all the other domain names. Second choice : Robots.txt with disallow for all the other websites. Third choice : I'm really open to other suggestions 😉 Thank you very much! 🙂
Intermediate & Advanced SEO | | Louis-Philippe_Dea0 -
Community question- Penguin 2.0 link types?
What type of links do you think Penguin 2.0 targeted most - anchor text abuse , directory links, paid links, low quality guest posts, article directories etc????
Intermediate & Advanced SEO | | DavidKonigsberg0
