Is Googlebot ignoring directives? Or is it Me?
-
I saw an answer to a question in this forum a few days ago, that said it was a bad idea to use robots.txt to tell googlebot to go away.
That SEO said it was much better to use the META tag to say noindex,nofollow.
So I removed the robots directive and added the META tag
<meta robots='noindex,nofollow'>
Today, I see google showing my send to a friend page where I expected the real page to be.
Does it mean Google is stupid?
Does it mean google ignores the Robots META tag?
Does it mean short pages have more value than long pages?
Does it mean if I convert my whole site to snippets, I'll get more traffic?
Does it mean garbage trumps content?
I have more questions, but this is more than enough.
-
Thank you Ryan.
They completely ignored the meta tags., completely messing up our serps. So I put it back in robots. I wont trust google again to do the right thing.
-
Hi Allan,
It is a best practice to use meta tags to indicate your indexing preference to search engines.
Normally the recommended implementation would be "noindex, follow" but without examining your site it is impossible to know for sure.
Google honors meta tags but there are a number of issues which could be the source of your issue. For example, if you did not use valid syntax the tag may not be honored. If you are blocking the page in robots.txt, then search engines cannot read the tag.
As for the last three questions, the simple answer is quality content is best.
If you can share the URL of the page involved, we can offer a specific response to the implementation of the meta tag.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Spam: how to filter out direct website visitors from Ukraine and Germany
Hi there, I was reading this article: https://moz.com/blog/stop-ghost-spam-in-google-analytics-with-one-filter as I am experiencing a lot of visitors from Ukraine and Germany as direct website visitors with only 1 pageview, a huge bounce rate (of course) and little time spend, so I am convinced its spam. In the article, it explains how to find the hostname to set up a filter but if I do so, I see that the highest amount of website visitors with this behavior come from the hostname that has my own domain name in it, so I can't exclude this. What should I do?
Technical SEO | | MarketingRumors0 -
Google ignoring the Title Tag?
Anybody seen this too? We have a webpage with tiny different title tag and H1. If you search for let's say "Renovatie", you get to see the title tag "De kostprijs van je renovatie". However, when you search with the search term "Wat kost een renovatie", we see the H1 title in the SERP, which is "Wat kost een renovatie". So that's normal when you search a term that's exact the same as the H1 tag, Google ignores the title tag? N.
Technical SEO | | nans0 -
How can I make it so that robots.txt is not ignored due to a URL re-direct?
Recently a site moved from blog.site.com to site.com/blog with an instruction like this one: /etc/httpd/conf.d/site_com.conf:94: ProxyPass /blog http://blog.site.com
Technical SEO | | rodelmo4
/etc/httpd/conf.d/site_com.conf:95: ProxyPassReverse /blog http://blog.site.com It's a Wordpress.org blog that was set as a subdomain, and now is being redirected to look like a directory. That said, the robots.txt file seems to be ignored by Google bot. There is a Disallow: /tag/ on that file to avoid "duplicate content" on the site. I have tried this before with other Wordpress subdomains and works like a charm, except for this time, in which the blog is rendered as a subdirectory. Any ideas why? Thanks!0 -
Please let me know if I am in a right direction with fixing rel="canonical" issue?
While doing my website crawl, I keep getting the message that I have tons of duplicated pages.
Technical SEO | | kirupa
http://example.com/index.php and http://www.example.com/index.php are considered to be the duplicates. As I figured out this one: http://example.com/index.php is a canonical page, and I should point out this one: http://www.example.com/index.php to it. Could you please let me know if I will do a right thing if I put this piece of code into my index.php file?
? Or I should use this one:0 -
Is alt text inside an img tag inside an h1 the same weight as text directly inside the h1?
Right now I use a background image and CSS to tie the h1 tag to my logo on each page. However, I am concerned that may not be best practice. Plus, I am interested in using schema markup on my logo. So, my question is, if I use an image with alt text inside my h1 tag, will the alt text carry as much weight as a text-based h1?
Technical SEO | | Avalara0 -
Are my canonical re directs working?
Buonjourno from Wetherby UK 🙂 Ive been battlling sometime to get this site http://www.goldsboroughestates.co.uk to rank for term Right To Manage. Amongst other tactics ive set up a canonical http://www.goldsboroughestates.co.uk/about/right-to-manage.aspx * - Canonical version http://www.goldsboroughestates.co.uk/how-we-care-for-you/right-to-manage.aspx http://www.goldsboroughestates.co.uk/what-our-customers-say/right-to-manage.aspx http://www.goldsboroughestates.co.uk/ - But has this canonical redirect feature worked? The reason i doubt it is i notice when i enter a page http://www.goldsboroughestates.co.uk/what-our-customers-say/right-to-manage.aspx which has the below code in place: rell="canonical" href="http://www.goldsboroughestates.co.uk/About/right-to-manage.aspx/" /> It does not jump to http://www.goldsboroughestates.co.uk/about/right-to-manage.aspx So my question is... "is the canonical redirect working or not & waht is the best way / tool for diagnosing a canonical error" Grazie tanto,
Technical SEO | | Nightwing
David0 -
Has anyone seen direct improvement after April 23 by requesting reinclusion?
Using the open site explorer I have figured out that my former seo agency was buying name spam (mostly Asian sites)for my main keywords and did the same in a private network of blogs. I don't speak any eastern languages and seo Super Dude has left the planet. So... I don't really have much to report to the Google Webmaster folks. How much time - effort- cash do invest in removal requests vs, redo the whole darn site and hope for the best? All the best. Tom
Technical SEO | | tvw1300 -
Webmaster woes - should I re-direct or re-structure?
Hey guys, I'll get straight to the point - a small (growing) website I'm working on has a number links pointing to it from totally irrelevant sites (66, to be precise). These were built by an SEO company prior to me working on the site, and lead to an over-optimisation penalty for one keyword. This number doesn't sound large, but proportionally (to all other links), it is. It didn't used to be, but a lot of the links coming in have now 'died', and the domains they came from are now just parked. Anyway, I have managed to contact pretty much all the webmasters, and 27 of these links have been removed. Unfortunately - as I'm sure many people know all too well - a good handful of the contacted webmasters haven't replied, and the bad links still remain on their websites (either in-content or on links pages). I have decided to 'refresh' the website with some new (and better) content - providing much more information and a valuable resource. My question is - what should I do? Should I just replace the content on the existing pages (slightly altering the URL structure to match the topic more) and 301 the old URLs to the new ones? Or should I delete the pages and create new ones - thus making sure this particular section of the site isn't affected by any bad in-bound links? I'm more inclined to opt for the latter option, and 'start fresh' with the pages - so I know I've got total control over them, but wanted to get the opinion of the community before I made a decision. Thanks in advance for your responses! Nick
Technical SEO | | Danapollo0