Not using a robot command meta tag
-
Hi SEOmoz peeps.
Was doing some research on robot commands and found a couple major sites that are not using them.
If you check out the code for these:
http://www.zappos.com/product/7787787/color/92100
You fill not find a meta robot command line.
Of course you need the line for any noindex, nofollow, noarchive pages.
However for pages you want crawled and indexed, is there any benefit for not having the line at all?
Thanks!
-
Barry is spot on. You either have the meta robot line, or you do your robots.txt. I prefer to use the txt because it shows me all the rules I impose on the website at a glance and doesn't make me go and search each page individually if anything needs to be changed. Imagine having a website with 20000 pages where each is individually adjusted. If you suddenly need to change something it will take you forever. With a robots.txt it will literally take seconds.
-
Doesn't matter if the line's not there if you don't want the robots to do anything. Not including it just assumes the position that they can do whatever they're doing ie Index, Follow
All of those sites do have a few things in robots.txt though (and the meta is really for people that can't edit robots.txt)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should you use www?
The age old question. Should I use "www." for a brand new content site assuming my goal (and most goals starting out) is to get to millions of visits per month? Does the community agree with, http://www.yes-www.org/why-use-www/? The only reason I question it honestly, since most high traffic companies in my search use www., is because moz doesn't. Thanks for your help. Seems it would be quite a pain to go back once you have a lot of traffic.
Intermediate & Advanced SEO | | mag7770 -
URLs with parameters + canonicals + meta robots
Hi Moz community! I'm posting a new question here as I couldn't find specific answer to the case I'm facing. Along with canonical tags, we are implementing meta robots on our pages (e-commerce website with thousands of pages). Most of the cases have been covered but I still have one unanswered case: our products are linked from list pages (mostly categories) but they almost always include a tracking parameter (ie /my-product.html?ref=xxx) products urls are secured with a canonical tag (referring only to the clean url /my-product.html) but what would be the best solution regarding the meta robots? For now we opted for a meta robot 'noindex, follow' for non canonical urls (so the ones unfortunately linked from our category/list pages), but I'm afraid that it could hurt our SEO (apparently no juice is given from URLs with a noindex robots), and even maybe prevent bots from crawling our website properly ... Would it be best to have no meta robots at all on these product urls with parameters? (we obviously can't have 'index, follow' when the canonical ref points to another url!). Thanks for your help!
Intermediate & Advanced SEO | | JessicaZylberberg0 -
Re-using content
Hi, I've just sold the domain for a website, so I'm free to re-purpose the content over to another website I own. How can I make sure that Gg doesn't deem it as duplicate? Do I need to let Gg naturally realise that the 'original' website no longer has the content on it? Do I need to hold-off putting the content live again? Should I notify Gg by-way of a de-index request, etc (assuming the domain won't incur any difficulty if I do this)? Thanks in advance.
Intermediate & Advanced SEO | | newstd1000 -
Why google index some meta titles I dont have?
Hi there, I have a problem with a website and I am desperate to find a solution because I have tried many things and nothing works! My website its: adtriboo.com Google does not find my main URL (main countro spain) www.adtriboo.com/es and I dont see this page its indexed in google. See link https://www.google.es/search?num=100&hl=es&site=&source=hp&q=site%3Aadtriboo.com&oq=site%3Aadtriboo.com&gs_l=hp.3...1189.4419.0.4586.17.17.0.0.0.0.223.1457.9j6j1.16.0...0.0...1c.1.8.hp.brTKX-zPwVI Also, google its showing some meta titles that are not in my page! For example my subfolder for the country Chile shows this title: Chile - Adtriboo but this its my real title Diseño logo, logotipos, video corporativo - adtriboo In webmaster tools everything looks good, and if I explore the webpage like google in webmaster tools the code its ok and everything lookd okay. If you see for example the URL from Chile (www.adtriboo.com/es_CL) the meta title is not the right one! Also i have a problem indexatión because i am not visible for any of my keywords even in the page 10! Please, somebody knows what happen?
Intermediate & Advanced SEO | | Comunicare0 -
Title tags not displaying correctly?
If I have a title tag with 28 words (around 90 letters) with some repetition and Google has decided not to use this in the SERPS is this a bad thing (basically saying the title tag is not liked by Google).
Intermediate & Advanced SEO | | BobAnderson0 -
Block an entire subdomain with robots.txt?
Is it possible to block an entire subdomain with robots.txt? I write for a blog that has their root domain as well as a subdomain pointing to the exact same IP. Getting rid of the option is not an option so I'd like to explore other options to avoid duplicate content. Any ideas?
Intermediate & Advanced SEO | | kylesuss12 -
Blocking Dynamic URLs with Robots.txt
Background: My e-commerce site uses a lot of layered navigation and sorting links. While this is great for users, it ends up in a lot of URL variations of the same page being crawled by Google. For example, a standard category page: www.mysite.com/widgets.html ...which uses a "Price" layered navigation sidebar to filter products based on price also produces the following URLs which link to the same page: http://www.mysite.com/widgets.html?price=1%2C250 http://www.mysite.com/widgets.html?price=2%2C250 http://www.mysite.com/widgets.html?price=3%2C250 As there are literally thousands of these URL variations being indexed, so I'd like to use Robots.txt to disallow these variations. Question: Is this a wise thing to do? Or does Google take into account layered navigation links by default, and I don't need to worry. To implement, I was going to do the following in Robots.txt: User-agent: * Disallow: /*? Disallow: /*= ....which would prevent any dynamic URL with a '?" or '=' from being indexed. Is there a better way to do this, or is this a good solution? Thank you!
Intermediate & Advanced SEO | | AndrewY1 -
H1 Tags
Quick and easy most likely - Just need to clear a few point. I understand each page within the site should only have one H1 tag which should be the most important one. I also believe these only effect google ranking very slightly? right? Currently my CMS is system is pulling the H1 tag in from the page and automatically using the page heading that is on the page IE) the heading used for the content. Should this be a keyword / key phrase instead? and will it be duplicate if i used the same one on various pages in my site? Cheers guys look forward to hearing your feedback
Intermediate & Advanced SEO | | wazza19850