Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to Configure Robots.txt File
How to configure correctly the Robots.txt File of the website. Need proper process to follow.Because a lot of my website URLs are excluded by google with the issues in the Robots.txt file.
International SEO | | seobac1 -
Does not having any hreflang tags for U.S Visitors lead to an increase in International Visitors?
I have seen a massive increase in International Visitors on our website and visitors within the United States dropped off hard this month (by about 20%). Could it be possible that not having any hreflang tags can lead to an increase in International Customers visiting the site even though your sitemap is set to "Target users in United States" within the Google Search Console? In the Google Search Console, I have International Targeting set to "Target users in United States." However, Google Search Console is saying our site doesn't have any hreflang tags. In the Google Search Console, it says "Your site has no hreflang tags. Google uses hreflang tags to match the user's language preference to the right variation of your pages." I'm not sure when that was flagged, but recently we have seen a massive increase in International Visitors to our site from countries such as Russia, Vietnam, Indonesia, the United Kingdom and so on. This poses a problem since our chances of turning one of those visitors into a customer is extremely slim. Along with that, nearly every international customer is contributing to an extremely high Bounce Rate. Attached is a screenshot of the Error about hreflang tags. https://imgur.com/a/XZI45Pw And here is a screenshot of the Country we are targeting. https://imgur.com/a/ArpWe9Z Lastly, attached is a screenshot of all of the Countries that visited our site today: https://imgur.com/a/d0tNwkI
International SEO | | MichaelAtMSP1 -
Google Analytics - whats best way to set up to track users on geotargeted sub folders & tlds ?
Hi ive got a client site just about to launch into 2 new territories (German Germany & Turkish Turkey) The existing/main site is in English & targets English keyword usage globally via domain.com and is not geotargeted in anyway. Analytics is set up for this on domain.com and tracking accurately and GWT is set up too. Now we have added a Turkish language site via a subfolder (domain.com/tr/) & the German site (domsain.com/de/) BUT this German subfolder will be domain mapped to a TLD (domain.de) eventually.. So what's best way to configure/set up Google Analytics to track users so can view the international sub-sites as individual sites etc ? (& GWT for that matter) ? Also after doing this i will be setting up in Moz Anlytics as separate campaigns so how do you set up so only draws in the Analytics data specific to these sub-folders ? All Best Dan
International SEO | | Dan-Lawrence0 -
Mini Google Search box within organic sitelinks
I'm trying to find solid information on Google's criteria on showing a "mini Google search box" within sitelinks? Example: if you type-in "wikipedia" on Google.com, a mini search box appears within the site links. This is for organic listing not paid listing. The Google Support page on Sitelinks does not state any info on the mini google search box. http://support.google.com/webmasters/bin/answer.py?hl=en&answer=47334
International SEO | | Mobies0 -
How long does it take for google to realize there is a new language sub domain?
I have a global client who just launched new sub domains of their site in new languages (Spanish, Portuguese, Russian and Japanese). Example: es.client.com pt.client.com ru.amdocs.com ja.amdocs.com How long does it take for Google to recognize these new subdomains? Would submitting an XML sitemap be helpful? If so how long can we see them pick up the site? What is the best way to check that your new subdomains are showing in each countries SERPs? Thanks for your help!
International SEO | | Scratch_MM0 -
Google Webmaster Tools - Geographic target - Time till change
One of my clients is targeting UK customers. They have UK IP address. I then noticed that they had their Geographic target set to "Target users in: United States". I advised them to change this. Anyone know: How this could have been set, the client says they didn't do it? How long will it take for Google to see this has been changed? Thanks in advance. Justin
International SEO | | GrouchyKids0 -
Non US site pages indexed in US Google search
Hi, We are having a global site wide issue with non US site pages being indexed by Google and served up in US search results. Conversley, we have US en pages showing in the Japan Google search results. We currently us IP detect to direct users to the correct regional site but it isn't effective if the users are entering through an incorrect regional page. At the top of each or our pages we have a drop down menu to allow users to manually select their preferred region. Is it possible that Google Bot is crawling these links and indexing these other regional pages as US and not detecting it due to our URL structure? Below are examples of two of our URLs for reference - one from Canada, the other from the US /ca/en/prod4130078/2500058/catalog50008/ /us/en/prod4130078/2500058/catalog20038/ If that is, in fact, what is happening, would setting the links within the drop down to 'no follow' address the problem? Thank you. Angie
International SEO | | Corel0 -
Google Webmaster Tools - International SEO Geo-Targeting site with Worldwide rankings
I have a client who already has rankings in the US & internationally. The site is broken down like this: url.com (main site with USA & International Rankings) url.com/de url.com/de-english url.com/ng url.com/au url.com/ch url.com/ch-french url.com/etc Each folder has it's own sitmap & relative content for it's respective country. I am reading in google webmaster tools > site config > settings, the option under 'Learn More': "If you don't want your site associated with any location, select Unlisted." If I want to keep my client's international rankings the way it currently is on url.com, do NOT geo target to United States? So I select unlisted, right? Would I use geo targeting on the url.com/de, url.com/de-english, url.com/ng, url.com/au and so on?
International SEO | | Francisco_Meza0