Is Googlebot ignoring directives? Or is it Me?
-
I saw an answer to a question in this forum a few days ago, that said it was a bad idea to use robots.txt to tell googlebot to go away.
That SEO said it was much better to use the META tag to say noindex,nofollow.
So I removed the robots directive and added the META tag
<meta robots='noindex,nofollow'>
Today, I see google showing my send to a friend page where I expected the real page to be.
Does it mean Google is stupid?
Does it mean google ignores the Robots META tag?
Does it mean short pages have more value than long pages?
Does it mean if I convert my whole site to snippets, I'll get more traffic?
Does it mean garbage trumps content?
I have more questions, but this is more than enough.
-
Thank you Ryan.
They completely ignored the meta tags., completely messing up our serps. So I put it back in robots. I wont trust google again to do the right thing.
-
Hi Allan,
It is a best practice to use meta tags to indicate your indexing preference to search engines.
Normally the recommended implementation would be "noindex, follow" but without examining your site it is impossible to know for sure.
Google honors meta tags but there are a number of issues which could be the source of your issue. For example, if you did not use valid syntax the tag may not be honored. If you are blocking the page in robots.txt, then search engines cannot read the tag.
As for the last three questions, the simple answer is quality content is best.
If you can share the URL of the page involved, we can offer a specific response to the implementation of the meta tag.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Trackback URLs & temporary re-directs
Hi Community, I have receiving an increasing number of temporary re-direct status codes via MOZ crawl diagnostics. I have taken a look more closely and these URLs are 'trackback urls' from blog posts, the website is Wordpress integrated. What is best practice for these 302 temporary redirects? I have read that best practice for a 302 is to 301 re-direct a URL, but is this the case for a trackback URL?
Technical SEO | | SO_UK0 -
Googlebot size limit
Hi there, There is about 2.8 KB of java script above the content of our homepage. I know it isn't desirable, but is this something I need to be concerned about? Thanks,
Technical SEO | | SSFCU
Sarah Update: It's fine. Ran a Fetch as Google and it's rendering as it should be. I would delete my question if I could figure out how!0 -
NOFOLLOW Links: Can we 100% ignore them for SEO purposes?
Some SEO articles say we can completely ignore NoFollow links. Other articles say they still matter - but then are very vague on what they count for or against. So which is it really? I do realize that they can provide traffic, and for that they are worthwhile. But it is SEO I am asking about... The SEO purpose I am most concerned with is the Link Profile. Separating the Follows from the NoFollows often gives really different anchor text distributions. If they don't matter, why do MOZ and other SEO Analysis programs still include them in their standard reports? (I can see some benefit to having them as part of the in-depth reports) So what's your thoughts? Can we 100% ignore the NoFollows for our SEO analysis?
Technical SEO | | GregB1230 -
Canonicals being ignored
Hi, I've got a site that I'm working with that has 2 ways of viewing the same page - a property details page. Basically one version if the long version: /property/Edinburgh/Southside-Newington/6CN99V and the other just the short version with the code only on the end: /6cn99v There is a canonical in place from the short version to the long version, and the sitemap.xml only lists the long version HOWEVER - Google is indexing the short version in the majority of cases (not all but the majority). http://www.website.com/property/Edinburgh/Southside-Newington/6CN99V"> Obviously "www.website.com" contains the URL of the site itself. Any thoughts?
Technical SEO | | squarecat.ben0 -
When Should I Ignore the Error Crawl Report
I have a handful of pages listed in the Error Crawl Report, but the report isn't actually showing anything wrong with these pages. I am double checking the code on the site and also can't find anything. Should I just move on and ignore the Error Crawl Report for these few pages?
Technical SEO | | ChristinaRadisic0 -
Penguin 301 Re-Direct
Hi I already know the answer to this but wanted some backup. A friend of mine has asked me to take a look at his site. He was using an SEO company that saw him get hit hard by Penguin. He did the right thing and started all over with a new domain name. I've just noticed that the old domain name is actually setup to 301 to the new domain. This has to be a bad thing right?
Technical SEO | | brightonseorob0 -
Google SERPs and NoIndex directives.
We have pages that have been added to robots.txt as url patterns in DisAllow. Also, we have the meta noindex tags on the pages themselves. But we are finding the pages in index. I don't think they are higher up in the rankings and they don't have any descriptions, any previews or any cached pages. Why does Google show these pages? Could it be due to internal or external linking?
Technical SEO | | gaganc0 -
Why googlebot indexing one page, not the other?
Why googlebot indexing one page, not the other in the same conditions? In html sitemap, for example. We have 6 new pages with unique content. Googlebot immediately indexes only 2 pages, and then after sometime the remaining 4 pages. On what parameters the crawler decides to scan or not scan this page?
Technical SEO | | ATCnik0