Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Robots.txt issue for international websites

Intermediate & Advanced SEO

808

JinnatUlHasan last edited by

In Google.co.uk, our US based (abcd.com) is showing:

A description for this result is not available because of this site's robots.txt – learn more

But UK website (uk.abcd.com) is working properly. We would like to disappear .com result totally, if possible. How to fix it?

Thanks in advance.
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

Can you share any information about your robots.txt?
1 Reply Last reply
Reply Quote 0
JinnatUlHasan @KeriMorgret last edited by

My main problem is in the homepage. Both host similar type of products and brands.

You may check the screenshot. Sorry, I had to blanked out the text.

Thanks in advance.

8BwQKcv
1 Reply Last reply
Reply Quote 0
KeriMorgret last edited by

Is it showing that for every page, or only some pages? If so, which types of pages? What's the contents of your robots.txt file for the US site?
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Should I have multiple websites for my different brands or one main website with different tabs/areas?

My client creates apps. As well as the apps they create themselves, they have made some of their own that cover various different topics. Currently they have individual websites for each of these apps, and a website for their app making business. They are asking whether they should just have one website - their app building site, which also includes information about the two apps they've built themselves. My feeling is it's better to keep them separate. The app building site is trying to appeal to a B2B audience and gain business to build new apps. AppA is trying to help carehomes and carers to streamline their business, and AppB is trying to help workplace and employee welfare. Combining them all will mean lots of mixed messaging/keywords even if we have dedicated areas on the site. I also think it will limit how much content we can create on each without being completely overwhelming for the user. If we keep them all separate then we can have a very clear user journey. I would of course recommend having blog posts or some sort of landing page to link to AppA and AppB's websites. Thoughts? Thank you!
Intermediate & Advanced SEO | | WhitewallGlasgow

0
Penalty for duplicate content on the same website?

Is it possible to get a penalty for duplicate content on the same website? I have a old custom-built site with a large number of filters that are pre-generated for speed. Basically the only difference is the meta title and H1 tag, with a few text differences here and there. Obviously I could no-follow all the filter links but it would take an enormous amount of work. The site is performing well in the search. I'm trying to decide whether if there is a risk of a penalty, if not I'm loath to do anything in case it causes other issues.
Intermediate & Advanced SEO | | seoman10

0
Mobile Canonical Tag Issue

Hey so, For our site
we have the desktop version: www.site.com/product-name/product-code/ The mobile version www.site.com/mobile/product-name/product-code So...on the desktop version we'd have the following.. | | Now my question is, what do we do as far as canonicals on the actual mobile URL? Would it be this? | |
| | OR are we NOT supposed to have mobile canonical tags whatsoever since we've already added "rel alternate" ? Would like some clarificaiton. | | |
Intermediate & Advanced SEO | | ggpaul562

0
Should I have as few internal links as possible?

On most pages of my site i have a Quick Links section, which gives x3 cross sales links to other products, a newsletter sign up link, link to Blog, x4 links from images to surveys, newsletters, feedback etc. Will these links be hurting my optimal SEO juice between pages, should the number of internal links be kept to a minimum? My site is www.over50choices.co.uk if that helps. Thanks
Ash
Intermediate & Advanced SEO | | AshShep1

0
Should comments and feeds be disallowed in robots.txt?

Hi My robots file is currently set up as listed below. From an SEO point of view is it good to disallow feeds, rss and comments? I feel allowing comments would be a good thing because it's new content that may rank in the search engines as the comments left on my blog often refer to questions or companies folks are searching for more information on. And the comments are added regularly. What's your take? I'm also concerned about the /page being blocked. Not sure how that benefits my blog from an SEO point of view as well. Look forward to your feedback. Thanks. Eddy User-agent: Googlebot Crawl-delay: 10 Allow: /* User-agent: * Crawl-delay: 10 Disallow: /wp- Disallow: /feed/ Disallow: /trackback/ Disallow: /rss/ Disallow: /comments/feed/ Disallow: /page/ Disallow: /date/ Disallow: /comments/ # Allow Everything Allow: /*
Intermediate & Advanced SEO | | workathomecareers

0
Website.com/blog/post vs website.com/post

I have clients with Wordpress sites and clients with just a Wordpress blog on the back of website. The clients with entire Wordpress sites seem to be ranking better. Do you think the URL structure could have anything to do with it? Does having that extra /blog folder decrease any SEO effectiveness? Setting up a few new blogs now...
Intermediate & Advanced SEO | | PortlandGuy

0
Export Website into XML File

Hi, I am having an agency optimize the content on my sites. I need to create XML Schema before I export the content into XML. What is best way to export content including meta tags for an entire site along with the steps on how to?
Intermediate & Advanced SEO | | Melia

0
XML Sitemap instruction in robots.txt = Worth doing?

Hi fellow SEO's, Just a quick one, I was reading a few guides on Bing Webmaster tools and found that you can use the robots.txt file to point crawlers/bots to your XML sitemap (they don't look for it by default). I was just wondering if it would be worth creating a robots.txt file purely for the purpose of pointing bots to the XML sitemap? I've submitted it manually to Google and Bing webmaster tools but I was thinking more for the other bots (I.e. Mozbot, the SEOmoz bot?). Any thoughts would be appreciated! 🙂 Regards, Ash
Intermediate & Advanced SEO | | AshSEO2011

0