Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Robots.txt issue with indexation

International SEO

725

profesia Subscriber last edited by

Hello

i have a problem with one of the rules for robots.txt

i have a multilingual mutation of entire page on www.example.com/en/

I want to make indexable /allow/ the main page under /en/

but not indexable /disallow/ everything else under /en/*

Please help me how to write the rule.
1 Reply Last reply
Reply Quote 0
SEOKeith @profesia last edited by

Well put the rest of the content in a different directory then and disallow that, thats the only other solution I can think of...
1 Reply Last reply
Reply Quote 0
profesia Subscriber @SEOKeith last edited by

There is no option like

/en/index.html

The only adress where you can reach the english main page version is www.example.com/en/
1 Reply Last reply
Reply Quote 0
SEOKeith last edited by

Name the page you want indexing something and you can use the following:

Disallow: /en/

Allow: /en/index.html

Always test robots.txt in google webmaster tools.

Hope that helps,

Keith
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

International SEO setup issues canonical URL

My site is www.grocare.com for one region and in.grocare.com for another region. Both of them have the same content except the currency for particular regions. Someone told me that google will take the content as duplicate and not rank either. I have setup hreflang and targeted different regions for both in the search console. I read many article which say canonical urls need to be setup for international seo sites. But Im not sure how to setup canonical urls and whether they are the right way to go . i just don't want my content deranked. Now i have setup hreflang properly after asking the moz community itself. So im hoping to get some help with this query too. TIA
International SEO | | grocare

0
Backlinks issue

Hi guys So I have 3 top level domains - I'm seeing backlinks that point to each other. Is this something that I need to fix? I'm not too sure if this practice is ethical, but I'm not entirely sure how all the links are pointing towards each other and if this is something I will get pulled up for how do i go about fixing it? My 3 domains are as follows: zenory.com, zenory.com.au and zenory.co.nz I would appreciate some advice around this! Cheers
International SEO | | edward-may

0
International Sites - Sitemaps, Robots & Geolocating in WMT

Hi Guys, I have a site that has now been launched in the US having originally just been UK. In order to accommodate this, the website has been set-up using directories for each country. Example: domain.com/en-gb domain.com/en-us As the site was originally set-up for UK, the sitemap, robots file & Webmaster Tools account were added to the main domain. Example: domain.com/sitemap.xml domain.com/robots.txt The question is does this now need changing to make it specific for each country. Example: The sitemap and robots.txt for the UK would move to: domain.com/en-gb/sitemap.xml domain.com/en-gb/robots.txt and the US would have its own separate sitemap and robots.txt. Example : domain.com/en-us/sitemap.xml domain.com/en-us/robots.txt Also in order to Geolocate this in WMT would this need to be done for each directory version instead of the main domain? Currently the WMT account for the UK site is verified at www.domain.com, would this need reverifying at domain.com/en-gb? Any help would be appreciated! Thanks!
International SEO | | CarlWint

0
Understanding the "Index Status" Data Inside Google Webmaster Tools

Currently there are total 2,787 Articles added to my Blog. The Index Status shows the following report under Index Status>Advance Total Indexed = 12,505 Blocked by robots = 8,659 And when I do search for site:techmaish.com in Google.com, it shows; About 12,200 results (0.15 seconds) Now my question. 1:- Is it normal Or there is something wrong? 2:- If there is something wrong then what is that? Thanks in advance. _ Attached is the screenshot of my GWT._ 7dk.png
International SEO | | techmaish

0
Massive jump in pages indexed (and I do mean massive)

Hello mozzers, I have been working in SEO for a number of years but never seen anything like a jump in pages indexed of this proportion (image is from the Index Status report in Google Webmaster Tools: http://i.imgur.com/79mW6Jl.png Has anyone has ever seen anything like this?
Anyone have an idea about what happened? One thing that sprung to mind might be that the same pages are now getting indexed in several more google country sites (e.g. google.ca, google.co.uk, google.es, google.com.mx) but I don't know if the Index Status report in WMT works like that. A few notes to explain the context: It's an eCommerce website with service pages and around 9 different pages listing products. The site is small - only around 100 pages across three languages 1.5 months ago we migrated from three language subdomains to a single sub-domain with language directories. Before and after the migration I used hreflang tags across the board. We saw about 50% uplift in traffic from unbranded organic terms after the migration (although on day one it was more like +300%), especially from more language diversity. I had an issue where the 'sort' links on the product tables were giving rise to thousands of pages of duplicate content, although I had used the URL parameter handling to communicate to Google that these were not significantly different and only to index the representative URL. About 2 weeks ago I blocked them using the robots.txt (Disallow: *?sort). I never felt these were doing us too much harm in reality although many of them are indexed and can be found with a site:xxx.com search. At the same time as adding *?sort to the robots.txt, I made an hreflang sitemap for each language, and linked to them from an index sitemap and added these to WMT. I added some country specific alternate URLs as well as language just to see if I started getting more traffic from those countries (e.g. xxx.com/es/ for Spanish, xxx.com/es/ for Spain, xxx.xom/es/ for Mexico etc). I dodn't seem to get any benefit from this. Webmaster tools profile is for a URL that is the root domain xxx.com. We have a lot of other subdomains, including a blog that is far bigger than our main site. But looking at the Search Queries report, all the pages listed are on the core website so I don't think it is the blog pages etc. I have seen a couple of good days in terms of unbranded organic search referrals - no spike or drop off but a couple of good days in keeping with recent improvements in these kinds of referrals. We have some software mirror sub domains that are duplicated across two website: xxx.mirror.xxx.com and xxx.mirror.xxx.ca. Many of these don't even have sections and Google seemed to be handling the duplication, always preferring to show the .com URL despite no cross-site canonicals in place. Very interesting, I'm sure you will agree! THANKS FOR READING! 79mW6Jl.png
International SEO | | Lina-iWeb

0
Multinational Sites - The main SEO issues

I currently work for the UK arm of a Company with headquarters in Germany - The have outlets in half-a-dozen European countries, and up until now each country has had it's own website. The group has decided that from next year they will close all the individual country sites and then run new sites each from a central .location, I guess with a shared database of products. I see the sense in having central stock control etc, but I'm worried about the SEO impact. I have searched Q&A and the blog but could not find much to help me. What I would like to do is to provide some advice and pointers at to what they should be aiming for, both in terms website structure and on-going SEO for each country. Any advice welcome, thanks in advance.
International SEO | | cottamg

0
Google Indexing Part two

Hi Everybody, I am trying to understand how does Google works, so I ve been reading and researching a lot. But I am still having a problem that I cannot solve. My website is in several languages, but its main language is Catalan. so if you get into my webite: "www.vallnord.com" the default language will be catalan. but if someone using Google.es in Spanish I would like the spanish version of the web to be the main result not the catalan. unfortunately this does not work like this. For a search query like "Esqui Andorra" the catalan version is on the 1st page and the spanish (www.vallnord.com/es) is on the 4th page. Does anybody know why is this happening or how can I solve it? Regards.
International SEO | | SilbertAd

0
Geotargetting Issues

I have a different problem then most. My international website (www.solmelia.com) is showing number one in english for "sol melia" in the Mexican google search engine. Plus the 3rd listing on google.com.mx is our homepage in spanish but it is showing up as a 401. We need to redirect the ccTLD (www.solmelia.es) to our current spanish version that is actually a subdomain (es.solmelia.com). Please let me know how I can fix both issues.
International SEO | | Melia

0