Why are these results being showed as blocked by robots.txt?
-
If you perform this search, you'll see all m. results are blocked by robots.txt: http://goo.gl/PRrlI, but when I reviewed the robots.txt file: http://goo.gl/Hly28, I didn't see anything specifying to block crawlers from these pages.
Any ideas why these are showing as blocked?
-
Hi,
Your robots.txt file is very .. steroid healthy. It has his own universe
Are you 100% sure all of the entries are legit and clean ?
First thing I would do is to check Web M;aster Tools for the mobile subdomain. If you don't have it yet, that will be a good place to start - to verify the m subdomain.
Once in WeB Master Tools - you can debug this in no time.
Cheers.
-
but, even when i search from my mobile device, I get the same results (that m. is blocked)
-
I can't submit because I haven't claimed m. in GWT
-
If you haven't already done so, I recommend testing your robots.txt file against one of your mobile pages (such as m.healthline.com/treatments) in Google Webmaster Tools. You can do this by logging into GWT, then click Health, then Blocked URLs.
If you have already tested it in GWT, can you let us know what the results said?
-
Another good article from the community
-
So after a little it or research as I never ever came past this before as all the site we do are responsive, I found this
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=72462
It seems Google wont index a website that they think is a mobile website within the main serp, and vice verse ...
Hope that helps, cause it had me puzzled
Regards
John
-
Which directory are you storing your mobile website files within ...
-
Oh, sorry, on further investigation I see its just your mobile site that are being blocked ...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is this on SERP results?
Hi Guys, Does anyone know what this is called: https://d.pr/i/RA6RsG And how Google pulls it from a page? Do you need some kind of markup? Cheers.
Intermediate & Advanced SEO | | nattyhall0 -
Tool to identify if meta description are showing?
Hi we have a Ecommerce client with 1000s of meta descriptions, we have noticed that some meta descriptions are not showing properly, we want to pull and see which ones are showing on Google SERP results. You can use tools like screaming frog to pull meta description from page, but we want to see if it's showing for certain keywords. Any ideas on how to automate this? Cheers.
Intermediate & Advanced SEO | | brianna00 -
Kinds of organic search results (Google)
Not sure if this is a new "unit" for Google organic results. Please see the attached image. When searching for "invoice software", the top quarter of the page is a ribbon of products/brands with badly formatted logos. The fact that it's so ugly, and there's nothing marking it as a paid result, leads me to think it's organic. Anyone know what this SERP unit is called; and better still: How do you get included? We rank super high in the normal organic results, but don't appear at all in this product ribbon. y71A9
Intermediate & Advanced SEO | | RobM4161 -
Incorrect URL shown in Google search results
Can anyone offer any advice on how Google might get the url which it displays in search results wrong? It currently appears for all pages as: <cite>www.domainname.com › Register › Login</cite> When the real url is nothing like this. It should be: www.domainname.com/product-type/product-name. This could obviously affect clickthroughs. Google has indexed around 3,000 urls on the site and they are all like this. There are links at the top of the page on the website itself which look like this: Register » Login » which presumably could be affecting it? Thanks in advance for any advice or help!
Intermediate & Advanced SEO | | Wagada0 -
Using Meta Header vs Robots.txt
Hey Mozzers, I am working on a site that has search-friendly parameters for their faceted navigation, however this makes it difficult to identify the parameters in a robots.txt file. I know that using the robots.txt file is highly recommended and powerful, but I am not sure how to do this when facets are using common words such as sizes. For example, a filtered url may look like www.website.com/category/brand/small.html Brand and size are both facets. Brand is a great filter, and size is very relevant for shoppers, but many products include "small" in the url, so it is tough to isolate that filter in the robots.txt. (I hope that makes sense). I am able to identify problematic pages and edit the Meta Head so I can add on any page that is causing these duplicate issues. My question is, is this a good idea? I want bots to crawl the facets, but indexing all of the facets causes duplicate issues. Thoughts?
Intermediate & Advanced SEO | | evan890 -
Robots.txt & Duplicate Content
In reviewing my crawl results I have 5666 pages of duplicate content. I believe this is because many of the indexed pages are just different ways to get to the same content. There is one primary culprit. It's a series of URL's related to CatalogSearch - for example; http://www.careerbags.com/catalogsearch/result/index/?q=Mobile I have 10074 of those links indexed according to my MOZ crawl. Of those 5349 are tagged as duplicate content. Another 4725 are not. Here are some additional sample links: http://www.careerbags.com/catalogsearch/result/index/?dir=desc&order=relevance&p=2&q=Amy
Intermediate & Advanced SEO | | Careerbags
http://www.careerbags.com/catalogsearch/result/index/?color=28&q=bellemonde
http://www.careerbags.com/catalogsearch/result/index/?cat=9&color=241&dir=asc&order=relevance&q=baggallini All of these links are just different ways of searching through our product catalog. My question is should we disallow - catalogsearch via the robots file? Are these links doing more harm than good?0 -
I have two sitemaps which partly duplicate - one is blocked by robots.txt but can't figure out why!
Hi, I've just found two sitemaps - one of them is .php and represents part of the site structure on the website. The second is a .txt file which lists every page on the website. The .txt file is blocked via robots exclusion protocol (which doesn't appear to be very logical as it's the only full sitemap). Any ideas why a developer might have done that?
Intermediate & Advanced SEO | | McTaggart0 -
Title tag showing in Google that we are not setting
Hello, We've noticed that when we do a specific search (print screen attached), that the business name and/or a completely different title is getting indexed into the search engine that we are not setting. Below is an example from the source code of how we're setting the title, this matches the 2nd listing circled in the attached image. The indexed title tag reflects "Animal Business Card Holders - Kyle Design" Any ideas or feedback on how this is happening? <title>Animal Business Card Cases in Pet, Insect and Wildlife Designstitle> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <meta name="description" content="Eye-catching business card holder cases personalized with custom animal designs for humane professionals and pet owners. Custom select a sleek metal finish, bold aluminum or iridescent accent color, size and unique design for the ultimate self-expressing animal gift!" /> <meta name="keywords" content="business card holder unique personalized custom holders silver gold wood metal cards cases sleek aluminum engraved contemporary case animal animals design designs black color accents iridescent pet insect wildlife cat dog dragonfly butterfly lions sea turtles sea otters elephants animal lover animal activist zoologist veterinarian breeder animal whisperer thin deep large credit Asian size engraving personalize gift gifts special monogram customized corporate logo name professional title meaningful sentiment" /> <meta name="copyright" content="Copyright Kyle Design" /> <meta name="author" content="Kyle Design" />
Intermediate & Advanced SEO | | marketing_zoovy.com
<meta name="generator" content="xyz Commerce System http://www.domain.com/" />
<link rel="canonical" href="xyz link"
<script type="text/javaScript"> Thanks,
Jamie0