I have two robots.txt pages for www and non-www version. Will that be a problem?
-
There are two robots.txt pages. One for www version and another for non-www version though I have moved to the non-www version.
-
It wont affect your SEO, you just don;t need the the non-https version
-
Hi ramb,
Short answer: No, it won't affect your ability to rank in Google. Unless both sites (non-www and www version) compete for the same search term and one of them isn't blocked in the correspondent robots.txt file.
If you can, make sure to have a redirection rule so as everything in the non-www goes to the www.
It bugs me why aren't you redirecting the complete non-www to the www version.
Two possibilities come to my mind:- You can't redirect the whole non-www due to some app or technical need.
In this case, both versions, if accessible to Google, will be treated as different sites. Thus, you must be sure that both robots file are correct for the given subdomain. - You have a separate website, which contains different content from the www version (this usually happens with subdomains with different page types, such as products.abc.com and categories.abc.com)
In this case, please be sure that you know what you want to be blocked and have each robots.txt file in their subdomain.
Keep in mind that Robots file only controls where you don't want googlebot to access in the public version of your website. When a certain page or group of pages are blocked in robots.txt, google won't access them anymore thus not knowing if that page has what it needs to rank for any given search term. Google might rank lower and users will see a note in search results, leading to a lower CTR.
Hope it helps.
Best Luck.
Gaston - You can't redirect the whole non-www due to some app or technical need.
-
Are you redirecting everything on www to non-www? If so, you don't really need a robots.txt to be served for the www subdomain. Google will ignore the original robots.txt file if it is given a 301 anyway.
-
Hi Gatson
Thank you for your response. Currently, www version of the site is redirected to non-www version, which is the primary(or root) domain.
But the problem is, I have 2 robots.txt files running for the same site. i.e. same robots.txt file loads on both www and non-www version. (Example https://www.abc.com/robots.txt and https://abc.com/robots.txt).
Does it affect my site's SEO ??
Should I redirect www-version of the file to non-www version?
Your feedback will be highly appreciated.Thank you,
R.
-
Hi ramb,
It's totally fine to have different robots.txt files for different subdomains.
Thus said, http://domain.com and http://www.domain.com are different subdomains. Consider the one with non-www as the full root domain.In case it is needed, here you have Google's official resource about robots.txt:
Learn about Robots.txt file - Search Console helpHope it helps.
Best luck.
Gast
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Role of Robots.txt and Search Console parameters settings
Hi, wondering if anyone can point me to resources or explain the difference between these two. If a site has url parameters disallowed in Robots.txt is it redundant to edit settings in Search Console parameters to anything other than "Let Googlebot Decide"?
Technical SEO | | LivDetrick0 -
Robots.txt Syntax for Dynamic URLs
I want to Disallow certain dynamic pages in robots.txt and am unsure of the proper syntax. The pages I want to disallow all include the string ?Page= Which is the proper syntax?
Technical SEO | | btreloar
Disallow: ?Page=
Disallow: ?Page=*
Disallow: ?Page=
Or something else?0 -
Pageing page and seo meta tag questions
Hi if i am using paging in my website there is lots of product in my website now in paging total paging is 1000 pages now what title tag i need to add for every paging page or is there any good way we can tell search engine all page or same ?
Technical SEO | | constructionhelpline0 -
Best use of robots.txt for "garbage" links from Joomla!
I recently started out on Seomoz and is trying to make some cleanup according to the campaign report i received. One of my biggest gripes is the point of "Dublicate Page Content". Right now im having over 200 pages with dublicate page content. Now.. This is triggerede because Seomoz have snagged up auto generated links from my site. My site has a "send to freind" feature, and every time someone wants to send a article or a product to a friend via email a pop-up appears. Now it seems like the pop-up pages has been snagged by the seomoz spider,however these pages is something i would never want to index in Google. So i just want to get rid of them. Now to my question I guess the best solution is to make a general rule via robots.txt, so that these pages is not indexed and considered by google at all. But, how do i do this? what should my syntax be? A lof of the links looks like this, but has different id numbers according to the product that is being send: http://mywebshop.dk/index.php?option=com_redshop&view=send_friend&pid=39&tmpl=component&Itemid=167 I guess i need a rule that grabs the following and makes google ignore links that contains this: view=send_friend
Technical SEO | | teleman0 -
Double problem: mobile friendly site and shopping cart page duplication
I have a website that has two issues related to SEO: 1) the main website (product.com) is not mobile-friendly and 2) I have a shopping cart site (buymyproduct.com) using Magento that basically duplicates our product pages that exist on the main marketing website. Uses click "buy now" on a product page and are sent to the checkout at "buymyproductnow.com". The company cannot overhaul product.com website right away and our shopping site (buymyproduct.com) uses a responsive theme and works well for iphone and iPad so I am thinking of making buymyproduct.com the mobile-friendly version of our website by using a sniffer on product.com and forwarding users to the mobile friendly version. If I add canonical references from the shopping cart product pages and articles back to product.com associated pages, will this lessen the blow to any seo issues? What other factors am I missing/need to consider Complicated and painful. Maybe doing nothing right now is best. Thanks for any feedback.
Technical SEO | | Timmmmy0 -
No indexing url including query string with Robots txt
Dear all, how can I block url/pages with query strings like page.html?dir=asc&order=name with robots txt? Thanks!
Technical SEO | | HMK-NL0 -
Wordpress Robots.txt Sitemap submission?
Alright, my question comes directly from this article by SEOmoz http://www.seomoz.org/learn-seo/r... Yes, I have submitted the sitemap to google, bing's webmaster tools and and I want to add the location of our site's sitemaps and does it mean that I erase everything in the robots.txt right now and replace it with? <code>User-agent: * Disallow: Sitemap: http://www.example.com/none-standard-location/sitemap.xml</code> <code>???</code> because Wordpress comes with some default disallows like wp-admin, trackback, plugins. I have also read this, but was wondering if this is the correct way to add sitemap on Wordpress Robots.txt. [http://www.seomoz.org/q/removing-...](http://www.seomoz.org/q/removing-robots-txt-on-wordpress-site-problem) I am using Multisite with Yoast plugin so I have more than one sitemap.xml to submit Do I erase everything in Robots.txt and replace it with how SEOmoz recommended? hmm that sounds not right. like <code> <code>
Technical SEO | | joony2008
<code>User-agent: *
Disallow: </code> Sitemap: http://www.example.com/sitemap_index.xml</code> <code>``` Sitemap: http://www.example.com/sub/sitemap_index.xml ```</code> <code>?????????</code> ```</code>0 -
Robots.txt file getting a 500 error - is this a problem?
Hello all! While doing some routine health checks on a few of our client sites, I spotted that a new client of ours - who's website was not designed built by us - is returning a 500 internal server error when I try to look at the robots.txt file. As we don't host / maintain their site, I would have to go through their head office to get this changed, which isn't a problem but I just wanted to check whether this error will actually be having a negative effect on their site / whether there's a benefit to getting this changed? Thanks in advance!
Technical SEO | | themegroup0