Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Robots.txt issue with indexation

International SEO

733

profesia Subscriber last edited by

Hello

i have a problem with one of the rules for robots.txt

i have a multilingual mutation of entire page on www.example.com/en/

I want to make indexable /allow/ the main page under /en/

but not indexable /disallow/ everything else under /en/*

Please help me how to write the rule.
1 Reply Last reply
Reply Quote 0
SEOKeith @profesia last edited by

Well put the rest of the content in a different directory then and disallow that, thats the only other solution I can think of...
1 Reply Last reply
Reply Quote 0
profesia Subscriber @SEOKeith last edited by

There is no option like

/en/index.html

The only adress where you can reach the english main page version is www.example.com/en/
1 Reply Last reply
Reply Quote 0
SEOKeith last edited by

Name the page you want indexing something and you can use the following:

Disallow: /en/

Allow: /en/index.html

Always test robots.txt in google webmaster tools.

Hope that helps,

Keith
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

International SEO setup issues canonical URL

My site is www.grocare.com for one region and in.grocare.com for another region. Both of them have the same content except the currency for particular regions. Someone told me that google will take the content as duplicate and not rank either. I have setup hreflang and targeted different regions for both in the search console. I read many article which say canonical urls need to be setup for international seo sites. But Im not sure how to setup canonical urls and whether they are the right way to go . i just don't want my content deranked. Now i have setup hreflang properly after asking the moz community itself. So im hoping to get some help with this query too. TIA
International SEO | | grocare

0
MultiRegional site indexing problems

Hello there!!! I have a multiregional site and dealing with some indexing problems. The problem is that google have only indexed our USA site We have: -set up hreflang tags -set up specific subdirectories https://www.website.com/ (en-us site and our main site) https://www.website.com/en-gb https://www.website.com/en-ca https://www.website.com/fr-ca https://www.website.com/fr-fr https://www.website.com/es-es ..... -set up automatic GEO IP redirects (301 redirects) -created a sitemap index and a different sitemap for each regional site -created a google webmaster's tool for each country targeted -created translations for each different language and added some canonicals to the US' site when using English content. The problem is that Google is not indexing our regional sites. I think that the problem is that google is using a US bot when spidering the site, so it will be always redirect to the US version by a 301 redirect. I have used fetch as google with some of our regional folders and asked for "Indexing requested for URL and linked pages", but still waiting. Some ideas?? changing 301 to 302? Really don't know what to do. Thank you so much!!
International SEO | | Alejandrodurn

0
.fr Site Not Indexed in Search Console

Hey everyone, I am just now learning international SEO, and so appreciate any and all help! 1. I added supershuttle.fr to Search Console about a week ago and it still has 0 indexed pages. It is showing that pages were indexed in the sitemap, and when I check, there are 75 results in Google. Is this something I should be concerned with? Is there a setting that I'm not aware of in Search Console that I need to change? 2. Also, I read the below regarding the automatically translated pages. Would https://en.supershuttle.fr/ be considered an "automatically translated" page? Use robots.txt to block search engines from crawling automatically translated pages on your site. Automated translations don’t always make sense and could be viewed as spam. More importantly, a poor or artificial-sounding translation can harm your site’s perception
International SEO | | SuperShuttle

0
Why Google is not indexing each country/language subfolder on the ranks?

Hi folks, We use Magento 2 for the multi-country shops (its a multistore). The URL: www.avarcas.com The first days Google indexed the proper url in each country: avarcas.com/uk avarcas.com/de ... Some days later, all the countries are just indexing / (the root). I correctly set the subfolders in Webmaster tools. What's happening? Thanks
International SEO | | administratorwibee

0
Lost local organic rankings and international issues

Hi Everyone, Hoping we can get some help from our fellow Mozzers (Mozee's? Mozites?) We have 3 TLD's .com.au .co.uk & .com We noticed an issue a couple of weeks ago where we suddenly lost a lot of our search rankings for keywords on the .com.au site that we'd been top with for a long time. A lot of our Australian visitors were coming through our US site. US & UK sites got increased ranking results. We fixed up what we thought were the issues (Potentially HREF Lang issues and old sitemap issues). Google Search Console is still telling us we have some HREF Lang Errors, (but this could be waiting an updated crawl as the number is decreasing) Our main domain example.com is now showing up as first result in google.com.au search and the example.com.au doesn't show up until page 4 (prior to 2 weeks ago it was number 1) Any input would be appreciated...
International SEO | | tinyme

0
Understanding the "Index Status" Data Inside Google Webmaster Tools

Currently there are total 2,787 Articles added to my Blog. The Index Status shows the following report under Index Status>Advance Total Indexed = 12,505 Blocked by robots = 8,659 And when I do search for site:techmaish.com in Google.com, it shows; About 12,200 results (0.15 seconds) Now my question. 1:- Is it normal Or there is something wrong? 2:- If there is something wrong then what is that? Thanks in advance. _ Attached is the screenshot of my GWT._ 7dk.png
International SEO | | techmaish

0
Ranking issues for UK vs US spelling - advice please

Hi guys, I'm reaching out here for what may seem to be a very simple and obvious issue, but not something I can find a good answer for. We have a .com site hosted in Germany that serves our worldwide audience. The site is in English, but our business language is British (UK) English. This means that we rank very well for (e.g.) optimisation software but optimization software is nowhere to be found. The cause of this to me seems obvious; a robot reading those two phrases sees two distinct words. Nonetheless, having seen discussions of a similar nature around the use of plurals in keywords, it would seem to me that Google should have this sort of thing covered. Am I right or wrong here? If I'm wrong, then what are my options? I really don't want to have to make a copy of the entire site; apart from the additional effort involved in content upkeep I see this path fraught with duplicate content issues. Any help is very much appreciated, thanks.
International SEO | | StevenHowe

0
Geolocation and Indexing

Hi all, Our company owns site that have over 5 millions pages in Google index. We are locating in German, but our business aimed to US market. So, recently I checked index of our site using region targeting in US and there were only 150k of pages, but when I checked targeting in German there were almost 5 billion pages. Our server/IP locating in US, all the backlinks are from US sites. So, why there it is only small part of the site indexed in US? Regards, Dmitry
International SEO | | bubliki

0