Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Robots.txt: Can you put a /* wildcard in the middle of a URL?

Intermediate & Advanced SEO

936

IHSwebsite last edited by

We have noticed that Google is indexing the language/country directory versions of directories we have disallowed in our robots.txt.

For example:

Disallow: /images/ is blocked just fine

However, once you add our /en/uk/ directory in front of it, there are dozens of pages indexed.

The question is: Can I put a wildcard in the middle of the string, ex. /en/*/images/, or do I need to list out every single country for every language in the robots file. Anyone know of any workarounds?
1 Reply Last reply
Reply Quote 0
irvingw last edited by

Yes, wildcards work, thank god.
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Changing URLs

URLs of my web pages are based on the titles of pages. For sampel, if a title page is called "product ABC", then the URL for this page is /product-abc. Google and all other search engines have indexed all pages. Now I want to change the titles of some sites. Should I change the URLs accordingly, or should I rather leave URLs as they are. SEO Best Practice says that keywords must be placed both in the title, and in the URL. I think that Google will think that pages have douplicate content with diffrent titles, and it comes to many 404 error, if I change the URLs. What do you recommend in this case?
Intermediate & Advanced SEO | | kian_moz

0
When the site's entire URL structure changed, should we update the inbound links built pointing to the old URLs?

We're changing our website's URL structures, this means all our site URLs will be changed. After this is done, do we need to update the old inbound external links to point to the new URLs? Yes the old URLs will be 301 redirected to the new URLs too. Many thanks!
Intermediate & Advanced SEO | | Jade

1
Question about Syntax in Robots.txt

So if I want to block any URL from being indexed that contains a particular parameter what is the best way to put this in the robots.txt file? Currently I have-
Disallow: /attachment_id Where "attachment_id" is the parameter. Problem is I still see these URL's indexed and this has been in the robots now for over a month. I am wondering if I should just do Disallow: attachment_id or Disallow: attachment_id= but figured I would ask you guys first. Thanks!
Intermediate & Advanced SEO | | DRSearchEngOpt

0
"noindex, follow" or "robots.txt" for thin content pages

Does anyone have any testing evidence what is better to use for pages with thin content, yet important pages to keep on a website? I am referring to content shared across multiple websites (such as e-commerce, real estate etc). Imagine a website with 300 high quality pages indexed and 5,000 thin product type pages, which are pages that would not generate relevant search traffic. Question goes: Does the interlinking value achieved by "noindex, follow" outweigh the negative of Google having to crawl all those "noindex" pages? With robots.txt one has Google's crawling focus on just the important pages that are indexed and that may give ranking a boost. Any experiments with insight to this would be great. I do get the story about "make the pages unique", "get customer reviews and comments" etc....but the above question is the important question here.
Intermediate & Advanced SEO | | khi5

0
URL categorization / subfolders

Hi Mozzers, We're currently in the process of a website redesign with new CMS and have the opportunity to change URL and structure. I would love some opinions as to what the best practise will be. A quick prerequisite, the website is entirely about France. French property, living, holidays, forum - everything. Therefore, we're unsure of the usage of the word France/French. Presently, we're running Classic ASP which allows for one subfolder then dynamic article ID. In my examples, I will take our activity holidays URL. At present this is /france-activity-holidays/DisplayArticle.asp?ID=12345. We know that DisplayArticle.asp?ID=12345 will simply become [article-title], however, its the preceding subfolders I would like some help with. Here are our thoughts on the options available. Can you please vote as to which you think is the best? /france-activity-holidays/ (one subfolder per category, as at present) /france/holidays/activity/ (always have a first subfolder with the word france) /holidays-to-france/activity-holidays/ (france in the primary subfolder) /holidays/activity-holidays-france/ (france in the secondary subfolder) /holidays/activity/ (because the whole website is about France, it is redundant to have /france/) /French-holidays/activity/ My gut feeling is either number 2 or 5. Concise, good for UX, OK for SEO. However, there is very little information around that is relevant to our sector. Thanks in advance! Matt
Intermediate & Advanced SEO | | Horizon

0
Renaming a URL

Hi, If we rename a URL (below) http://www.opentext.com/2/global/company/company-ecm-positioning.htm
to http://www.opentext.com/2/global/products/enterprise-content-management.htm (or something similar) Would search engines recognize that as a new page altogether? I know they would need to reindex it accordingly, so in theory it is kind of a "new" page. But the reason for doing this is to maintain the page's metrics (inbound links, authority, social activity, etc) instead of creating a new page from scratch. The page has been indexed highly in the past, so we want to keep it active but optimize it better and redirect other internal content (that's being phased out) to it to juice it up even more. Thanks in advance!
Greg
Intermediate & Advanced SEO | | pstables

0
Guest posts/article marketing can be considered as paid posts by SEs?

Hi, Guest posting/article Marketing is a major part of our link building strategy. Normally we get one or two links for our site and one or two links to other authority sites(relevant). Some time author bio is not published with article or even not mention that this is a guest post. ( I know that we get just link and missed the other advantages of guest if author bio isn't mention) Is this a good practice especially when some posts are published on low quality blogs? SEs can be considered these as paid posts? What will be the better way to do this? Thanks Alex
Intermediate & Advanced SEO | | alexgray

0
We are changing ?page= dynamic url's to /page/ static urls. Will this hurt the progress we have made with the pages using dynamic addresses?

Question about changing url from dynamic to static to improve SEO but concern about hurting progress made so far.
Intermediate & Advanced SEO | | h3counsel

0