Is Googlebot ignoring directives? Or is it Me?
-
I saw an answer to a question in this forum a few days ago, that said it was a bad idea to use robots.txt to tell googlebot to go away.
That SEO said it was much better to use the META tag to say noindex,nofollow.
So I removed the robots directive and added the META tag
<meta robots='noindex,nofollow'>Today, I see google showing my send to a friend page where I expected the real page to be.
Does it mean Google is stupid?
Does it mean google ignores the Robots META tag?
Does it mean short pages have more value than long pages?
Does it mean if I convert my whole site to snippets, I'll get more traffic?
Does it mean garbage trumps content?
I have more questions, but this is more than enough.
-
Thank you Ryan.
They completely ignored the meta tags., completely messing up our serps. So I put it back in robots. I wont trust google again to do the right thing.
-
Hi Allan,
It is a best practice to use meta tags to indicate your indexing preference to search engines.
Normally the recommended implementation would be "noindex, follow" but without examining your site it is impossible to know for sure.
Google honors meta tags but there are a number of issues which could be the source of your issue. For example, if you did not use valid syntax the tag may not be honored. If you are blocking the page in robots.txt, then search engines cannot read the tag.
As for the last three questions, the simple answer is quality content is best.
If you can share the URL of the page involved, we can offer a specific response to the implementation of the meta tag.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Please let me know if I am in a right direction with fixing rel="canonical" issue?
While doing my website crawl, I keep getting the message that I have tons of duplicated pages.
Technical SEO | | kirupa
http://example.com/index.php and http://www.example.com/index.php are considered to be the duplicates. As I figured out this one: http://example.com/index.php is a canonical page, and I should point out this one: http://www.example.com/index.php to it. Could you please let me know if I will do a right thing if I put this piece of code into my index.php file?
? Or I should use this one:0 -
NOFOLLOW Links: Can we 100% ignore them for SEO purposes?
Some SEO articles say we can completely ignore NoFollow links. Other articles say they still matter - but then are very vague on what they count for or against. So which is it really? I do realize that they can provide traffic, and for that they are worthwhile. But it is SEO I am asking about... The SEO purpose I am most concerned with is the Link Profile. Separating the Follows from the NoFollows often gives really different anchor text distributions. If they don't matter, why do MOZ and other SEO Analysis programs still include them in their standard reports? (I can see some benefit to having them as part of the in-depth reports) So what's your thoughts? Can we 100% ignore the NoFollows for our SEO analysis?
Technical SEO | | GregB1230 -
Will a google map loaded "on scroll" be ignored by the crawler?
One of my pages has two Google maps on it. This leads to a fairly high keyword density for words like "data", "map data" etc. Since one of the maps is basically at the bottom of the page I thought of loading it "on scroll" as soon as its container becomes visible (before loading the map div should be empty). Will the map then still be craweld by google (can they execute the JS in a way that the map is loaded anyways?) or would this help to reduce the keywords introduced by the maps?
Technical SEO | | ddspg0 -
Why can't i get the page if i type/paste url directly?
Hello, just click the following link, http://www.tuscany-cooking-class.com/es/alojamiento/villa-pandolfini/ It might be show the 404 page, but follow this way, www.tuscany-cooking-class.com/es then select alojamiento link, then select first property name with villa-pandolfini, Now you can view the page content, why it behave like this, We are using joomla with customized. Anyone help me to fix this issue Thanks Advance Alex
Technical SEO | | massimobrogi0 -
301 re-directs to similar content, good idea?
We need to remove about 50 articles from our site, and we can find similar content to re-direct these to. Would it be a good or bad idea to re-direct this content? Or should we just 404 it? Should a 301 re-direct always be an exact match?
Technical SEO | | nicole.healthline0 -
Why is google ignoring my sitelinks demotions?
I'm referring to the sitelinks that appear in the SERPs when searching for my brand name. 6 subpages come up- some found in my main navigation, some not. 3 of the 6 sitelinks have been demoted in Webmaster Tools under "Site Configuration > Sitelinks." I realize that they say in their instructions: "Google doesn't guarantee that demoted URLs will never appear as a sitelink, but we do consider a demotion a strong hint that we'll try to honor when generating sitelinks." I am just surprised they would ignore 3 of the demotions and for so long. The pages that are demoted do not have very many internal links pointing to them (unlike other pages of my site that are targeted specifically). Also, the site has tens of thousands of pages to choose from. Why are they ignoring my request? What else can I do to fix this?
Technical SEO | | Hakkasan0 -
If a page isn't linked to or directly sumitted to a search engine can it get indexed?
Hey Guys, I'm curious if there are ways a page can get indexed even if the page isn't linked to or hasn't been submitted to a search engine. To my knowledge the following page on our website is not linked to and we definitely didn't submit it to Google - but it's currently indexed: <cite>takelessons.com/admin.php/adminJobPosition/corp</cite> Anyone have any ideas as to why or how this could have happened? Hopefully I'm missing something obvious 🙂 Thanks, Jon
Technical SEO | | TakeLessons0 -
If I 301 re-direct a piece of content (A) to another piece of content (B) and B is unrelated in subject matter to A, will the referring search keywords to content piece A hold for content piece B?
For example, I have a piece of content about furniture and it ranks in top 5 in the SERPs for the phrase "furniture". If I were to 301 redirect that piece of furniture content to a piece of content about trucks, would the referring keyword "furniture" continue to rank over time for the trucks content? My instincts tell me that in the short term the content piece about trucks would receive traffic for the term "furniture", but over the long term, the trucks content would lose rankings for the term "furniture" since the piece has to do with trucks and not furniture. Any thoughts?
Technical SEO | | pbrothers240