Why are these results being showed as blocked by robots.txt?
-
If you perform this search, you'll see all m. results are blocked by robots.txt: http://goo.gl/PRrlI, but when I reviewed the robots.txt file: http://goo.gl/Hly28, I didn't see anything specifying to block crawlers from these pages.
Any ideas why these are showing as blocked?
-
Hi,
Your robots.txt file is very .. steroid healthy. It has his own universe
Are you 100% sure all of the entries are legit and clean ?
First thing I would do is to check Web M;aster Tools for the mobile subdomain. If you don't have it yet, that will be a good place to start - to verify the m subdomain.
Once in WeB Master Tools - you can debug this in no time.
Cheers.
-
but, even when i search from my mobile device, I get the same results (that m. is blocked)
-
I can't submit because I haven't claimed m. in GWT
-
If you haven't already done so, I recommend testing your robots.txt file against one of your mobile pages (such as m.healthline.com/treatments) in Google Webmaster Tools. You can do this by logging into GWT, then click Health, then Blocked URLs.
If you have already tested it in GWT, can you let us know what the results said?
-
Another good article from the community
-
So after a little it or research as I never ever came past this before as all the site we do are responsive, I found this
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=72462
It seems Google wont index a website that they think is a mobile website within the main serp, and vice verse ...
Hope that helps, cause it had me puzzled
Regards
John
-
Which directory are you storing your mobile website files within ...
-
Oh, sorry, on further investigation I see its just your mobile site that are being blocked ...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
A grade optimised posts not showing in SERPs
Hi all, I've been using Moz to research, optimise and grade a broad range of copy and blog posts over the years. After the optimisation process I've always seen a relatively quick improvement of pages/posts in SERPs. I am currently working on a new website launched earlier in the year on a subdomain. There's a sitemap, fresh content added every month and the site has an verified Google Analytics and Search Console account. The content is quite niche with low traffic data for related terms, however, I am finding that after three or four weeks the optimised posts aren't displaying in the top 50 results in Google. These are the posts: https://sykeshome.europe.sykes.com/cut-the-cost-of-customer-support-use-a-work-at-home-model/ - optimised for "Cut the cost of customer support" (and also "Cut the cost of customer support: use a work-at-home model") https://sykeshome.europe.sykes.com/quality-and-compliance-in-a-work-at-home-environment/ - optimised for "Quality and compliance" (and also "Quality and compliance in a work-at-home environment") As a new website launched on a subdomain there aren't currently any inbound links, but I wanted to know if I am simply being impatient in expecting the above posts to rank higher (if only slightly), or if there could be a reason optimised content with a Moz A grade isn't showing in the first 50 results. Any advice or pointers would be much appreciated. Jonathan
Intermediate & Advanced SEO | | JCN-SBWD0 -
Google stopped showing rich snippets - what does warning mean?
Hi, for our site prams.net, Google stopped showing the rich snippets yesterday. We got the following warning in Webmastertools "Data with spam structures http://www.prams.net/ may be removed from the search results" As we have not changed anything for months, I do not know what the reason could be. Here a sample url from our site http://www.prams.net/brevi-ovo-twin Does anybody have experience with this and can give me a tip what to change? Thanks in advance. Dieter Lang Here the full message. Google has found some of your pages markup with structures that violate our quality guidelines for structured data. To provide users with high quality search results, we show detailed search results only for content with markup that meet our quality guidelines. This manual action was against prams.net/ imposed. You should correct the markup and make a request for reconsideration. Having noted that the guidelines for the markup on the pages in question are complied with, we will remove this manual action. You can resolve this problem as follows: | 1 | Update your markup so that it no longer violates our guidelines for structured data Correct Incorrect markup or remove the markup that is not the content of your page properly. | See guidelines |
Intermediate & Advanced SEO | | Storesco
| 2 | Ask a reconsideration request Once you have corrected your website, you can submit a reconsideration request, so this manual action is canceled. Be specific so that we can understand what changes you have made to your site. | Reconsideration request | Need more help? | • | For more information on manual actions against data with spam structures can be found in our Help Center. |
| • | Refer to the section Guidelines for structured data about the problem. |
| • | Ask in our forum questions if you need more help. Make reference to the message type [WNC-632 900]. |0 -
Blank Cart Pages Showing as Duplicate, HELP
Hi Everyone, I'm seeing a bunch of URLs that look something like this [ domain.com/cart?add&id_product=42&token=776d4a08721f3d8c920e287248797547] showing as duplicate content in my Moz crawls. I think these are just blank pages for the most part. Is there anything to be concerned with here? Is there a way to clean this up? Thanks! Ricky
Intermediate & Advanced SEO | | RickyShockley0 -
To index search results or to not index search results?
What are your feelings about indexing search results? I know big brands can get away with it (yelp, ebay, etc). Apart from UGC, it seems like one of the best ways to capture long tail traffic at scale. If the search results offer valuable / engaging content, would you give it a go?
Intermediate & Advanced SEO | | nicole.healthline0 -
Do search results differ greatly when you search on mobile?
If you have a site with responsive design, is Google likely to look upon you more favourably and dramatically change rankings?
Intermediate & Advanced SEO | | BlueLinkERP0 -
Google Webmaster Now Shows YourMost Recent Links
I just saw this story today about a new Google Webmaster feature which lets you download a file of the most recent links. http://searchengineland.com/google-now-shows-you-your-most-recent-links-127903 I downloaded the file today and I already discovered a major site issue. Our site blog was completely duplicated on a secondary domain we own and Google was showing that site as recent links. I already emailed the dev team to fix this pronto. Anybody else using this new feature and perhaps can share if it helps you in any way.
Intermediate & Advanced SEO | | irvingw1 -
XML Sitemap instruction in robots.txt = Worth doing?
Hi fellow SEO's, Just a quick one, I was reading a few guides on Bing Webmaster tools and found that you can use the robots.txt file to point crawlers/bots to your XML sitemap (they don't look for it by default). I was just wondering if it would be worth creating a robots.txt file purely for the purpose of pointing bots to the XML sitemap? I've submitted it manually to Google and Bing webmaster tools but I was thinking more for the other bots (I.e. Mozbot, the SEOmoz bot?). Any thoughts would be appreciated! 🙂 Regards, Ash
Intermediate & Advanced SEO | | AshSEO20110 -
Subdomains - duplicate content - robots.txt
Our corporate site provides MLS data to users, with the end goal of generating leads. Each registered lead is assigned to an agent, essentially in a round robin fashion. However we also give each agent a domain of their choosing that points to our corporate website. The domain can be whatever they want, but upon loading it is immediately directed to a subdomain. For example, www.agentsmith.com would be redirected to agentsmith.corporatedomain.com. Finally, any leads generated from agentsmith.easystreetrealty-indy.com are always assigned to Agent Smith instead of the agent pool (by parsing the current host name). In order to avoid being penalized for duplicate content, any page that is viewed on one of the agent subdomains always has a canonical link pointing to the corporate host name (www.corporatedomain.com). The only content difference between our corporate site and an agent subdomain is the phone number and contact email address where applicable. Two questions: Can/should we use robots.txt or robot meta tags to tell crawlers to ignore these subdomains, but obviously not the corporate domain? If question 1 is yes, would it be better for SEO to do that, or leave it how it is?
Intermediate & Advanced SEO | | EasyStreet0