Robots.txt & Mobile Site
-
Background - Our mobile site is on the same domain as our main site. We use a folder approach for our mobile site abc.com/m/home.html We are re-directing traffic to our mobile site vie device detection and re-direction exists for a handful of pages of our site ie most of our pages do not redirect the user to a mobile equivalent page.
Issue – Our mobile pages are being indexed in desktop Google searches
Input Required – How should we modify our robots.txt so that the desktop google index does not index our mobile pages/urls
User-agent:
Googlebot-Mobile
Disallow: /m
User-agent: `YahooSeeker/M1A1-R2D2`
Disallow: /m
User-agent: `MSNBOT_Mobile`
Disallow: /m
Many thanks
-
Hi,
The fact that you do not have mobile versions for all pages of the site makes your case a bit of an odd one when it comes to best practices. I would recommend if at all possible to address this and make all pages available in mobile format also.
If for whatever reason this is not possible then I think you are better off using a meta robots tag NOINDEX to get your mobile results out of the serps completely and a rel canonical to the relevant desktop page to make extra sure there are no issues with duplicate content.
If you can get all pages into a mobile version then you will want to implement proper redirects based on user string as you are doing, and also implement rel alternate/canonical tags between the desktop and mobile pages of the site (which should make sure the right version of the site shows up in the relevant serps). For more details on the whole setup check:
http://moz.com/blog/how-to-optimize-a-mobile-site and
http://moz.com/blog/mobile-seo-process -
User-agent: Googlebot-Mobile
Disallow: /m
would't that statement prevent Googlebot-Mobile from crawling the mobile pages ? Why would you want to do that ?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site redesign makes Moz Site Crawl go haywire
I work for an agency. Recently, one of our clients decided to do a complete site redesign without giving us notice. Shortly after this happened, Moz Site Crawl reported a massive spike of issues, including but not limited to 4xx errors. However, in the weeks that followed, it seemed these 4xx errors would disappear and then a large number of new ones would appear afterward, which makes me think they're phantom errors (and looking at the referring URLs, I suspect as much because I can't find the offending URLs). Is there any reason why this would happen? Like, something wrong with the sitemap or robots.txt?
Technical SEO | | YYSeanBrady1 -
How do you link your adaptive mobile site to Google Analytics?
With Google now saying they're putting a lot more emphasis on mobile sites, we recently got notifications from Google Webmaster Tools saying that some of our pages are not built for mobile. Some of these pages, however have an adaptive page that when you visit from a mobile phone (m.mysite.com), you're taken to instead of the desktop version. My question is, how do I let Google know that I have an adaptive site and not get penalized for poor mobile usability? I already have Google Analytics on the mobile site, I just need to somehow let Webmaster tools / Google's web crawlers know that they should be looking to my mobile site for usability, not the desktop site. Any advice is appreciated!
Technical SEO | | Ditigal_Taylor0 -
Mobile Usability update mobile and desktop versions
Hello, A number of our clients have both m.website.com and website.com versions of their site. Per the latest update coming in April from Google, do I need to make sure all clients with just a desktop version are mobile-friendly (according to WT). Also, do I need to make both versions for those clients that have both m.website.com and website.com mobile-friendly or just the m.website.com? Thank you!!!!
Technical SEO | | lfrazer0 -
301 redirecting old content from one site to updated content on a different site
I have a client with two websites. Here are some details, sorry I can't be more specific! Their older site -- specific to one product -- has a very high DA and about 75K visits per month, 80% of which comes from search engines. Their newer site -- focused generally on the brand -- is their top priority. The content here is much better. The vast majority of visits are from referrals (mainly social channels and an email newsletter) and direct traffic. Search traffic is relatively low though. I really want to boost search traffic to site #2. And I'd like to piggy back off some of the search traffic from site #1. Here's my question: If a particular article on site #1 (that ranks very well) needs to be updated, what's the risk/reward of updating the content on site #2 instead and 301 redirecting the original post to the newer post on site #2? Part 2: There are dozens of posts on site #1 that can be improved and updated. Is there an extra risk (or diminishing returns) associated with doing this across many posts? Hope this makes sense. Thanks for your help!
Technical SEO | | djreich0 -
Mobile Theme Switching
Hi - wondering what one should do with a mobile theme in relationship to Google. We are using wordpress and then a plugin that detects whether or not the user is on a mobile device. Should you no-follow the mobile theme since it is effectively the same content as the main site? Or just leave it be and not worry about it. Thanks!
Technical SEO | | leeg0 -
Partial mobile sitemap
Hi, We have a main www website with a standard sitemap. We also have a m. site for mobile content (but m. is only for our top pages and doesn't include the entire site). If a mobile client accesses one of our www pages we redirect to the m. page. If we don't have a m. version we keep them on the www site. Currently we block robots from the mobile site. Since our m. site only contains the top pages, I'm trying to determine the boost we might get from creating a mobile sitemap. I don't want to create the "partial" mobile sitemap and somehow have it hurt our traffic. Here is my plan update m. pages to point rel canonical to appropriate www page (makes sure we don't dilute SEO across m. and www.) create mobile sitemap and allow all robots to access site. Our www pages already rank fairly highly so just want to verify if there are any concerns since m. is not a complete version of www?
Technical SEO | | NicB10