Robots.txt issue with indexation
-
Hello
i have a problem with one of the rules for robots.txt
i have a multilingual mutation of entire page on www.example.com/en/
I want to make indexable /allow/ the main page under /en/
but not indexable /disallow/ everything else under /en/*
Please help me how to write the rule.
-
Well put the rest of the content in a different directory then and disallow that, thats the only other solution I can think of...
-
There is no option like
/en/index.html
The only adress where you can reach the english main page version is www.example.com/en/
-
Name the page you want indexing something and you can use the following:
Disallow: /en/
Allow: /en/index.html
Always test robots.txt in google webmaster tools.
Hope that helps,
Keith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why are some regions/countries not indexing correctly?
Hi All, Recently I've added different regions (website.com/se/ etc) to Google search console and pointed them to their relevant countries, but only half are working when I search from a regions IP with a VPN and use the correct Google search ( Google.se etc ). Will this correct over time? or is something else causing them not to be indexed up correctly? Thanks in advance <colgroup><col width="81"><col width="104"></colgroup>
International SEO | | WattbikeSEO
| Country | Appear in SERP 17/12/2018 |
| AU | TRUE |
| CZ | TRUE |
| DK | TRUE |
| HK | TRUE |
| IE | TRUE |
| IT | TRUE |
| KR | TRUE |
| NL | TRUE |
| NZ | TRUE |
| SE | TRUE |
| SG | TRUE |
| US | TRUE |
| ZA | TRUE |
| AE | FALSE |
| AT | FALSE |
| CH | FALSE |
| CN | N/A |
| DE | FALSE |
| EE | FALSE |
| ES | FALSE |
| FI | FALSE |
| FR | FALSE |
| GB | FALSE |
| GR | FALSE |
| JP | FALSE |
| NO | FALSE |
| PL | FALSE |
| RU | FALSE |
| SI | FALSE |
| TR | FALSE |0 -
Another website clone issue
My site has been cloned by this f........ http://designer.aimeeprom.com/ original site http://www.5starweddingdirectory.com Still has our logo etc... How can we prevent this from happening, What should I do next. I have pinged them via the interactive chat but they do not reply..
International SEO | | Taiger0 -
International Sites - Sitemaps, Robots & Geolocating in WMT
Hi Guys, I have a site that has now been launched in the US having originally just been UK. In order to accommodate this, the website has been set-up using directories for each country. Example: domain.com/en-gb domain.com/en-us As the site was originally set-up for UK, the sitemap, robots file & Webmaster Tools account were added to the main domain. Example: domain.com/sitemap.xml domain.com/robots.txt The question is does this now need changing to make it specific for each country. Example: The sitemap and robots.txt for the UK would move to: domain.com/en-gb/sitemap.xml domain.com/en-gb/robots.txt and the US would have its own separate sitemap and robots.txt. Example : domain.com/en-us/sitemap.xml domain.com/en-us/robots.txt Also in order to Geolocate this in WMT would this need to be done for each directory version instead of the main domain? Currently the WMT account for the UK site is verified at www.domain.com, would this need reverifying at domain.com/en-gb? Any help would be appreciated! Thanks!
International SEO | | CarlWint0 -
Blocking domestic Google's in Robots.txt
Hey, I want to block Google.co.uk from crawling a site but want Google.de to crawl it. I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers? any ideas? Thanks B
International SEO | | Bush_JSM0 -
Do I have duplicate content issues to be worried about?
Hey guys, We built a website http://www.cylon.com/ targeting different regions but with the same English langauage (Ireland, England and America). The content for the most part is the same set up on 3 different subfolders. http://www.cylon.com/ - Targeting United States in WMT http://www.cylon.com/ie - Targeting Ireland in WMT http://www.cylon.com/uk - Targeting UK in WMT Do I have duplicate content issues to be worried about? If so, how do I get around this issue? Also is there anyway of finding out if Google have in some way penalised these pages for having the same content on other pages trageting different Countries? I have not received any messages from Google in WMT saying there is duplicate so I'm not sure if this is an issue. Thanks Rob
International SEO | | daracreative0 -
Geo targeting issue and hosting
Hi guys and gals, this is not a problem per se, but an oddity that I would appreciate some insight on from the big juicy brains in this community. Our site had hosting in the US, and I was concerned that therefore our relevance to our own country (Australia) was diminished because of it. For one of our main keywords we were a few spots behind the competitor on the 1st page for an australian searcher, but when i searched the same keyword from Google.com with gl=us to show US only results, we outranked the competitors by a few spots. On page elements aside (if anything we had more geo identifiers on the ranking page in question) I wanted to move hosts anyway and got hosting in Australia. The next week our search traffic jumped by 25%. But it was almost all US traffic. Australian traffic was unchanged. Any idea how this could happen? It's an .AU domain, hosted in Australia, with on page clearly identifying Australia. I checked webmaster tools and our geo is properly set to Australia. I checked the keywords that the traffic increased for and they are not geo specific at all. Besides that I don't know how else to pin this down. Thanks.
International SEO | | Digital3600 -
Geo Targeting for Similar Sites to Specific Countries in Google's Index
I was hoping Webmaster Tools geo targeting would prevent this - I'm seeing in select google searches several pages indexed from our Australian website. Both sites have unique TLDs: barraguard.com barraguard.com.au I've attached a screenshot as an example. The sites are both hosted here in the U.S. at our data center. Are there any other methods for preventing Google and other search engines from indexing the barraguard.com.au pages in searches that take place in the U.S.? dSzoh.jpg
International SEO | | longbeachjamie0 -
Is duplicate content really an issue on different International Google engines?
i.e. Google.com v.s. Google.co.uk This relates to another question I have open on a similar issue. So if I open the same e-commerce site (virtually) on company.com and company.co.uk, does Google really view that as duplicate content? I would be inclined to think they have that figured out but I havent had much experience with international SEO...
International SEO | | BlinkWeb0