Not using a robot command meta tag
-
Hi SEOmoz peeps.
Was doing some research on robot commands and found a couple major sites that are not using them.
If you check out the code for these:
http://www.zappos.com/product/7787787/color/92100
You fill not find a meta robot command line.
Of course you need the line for any noindex, nofollow, noarchive pages.
However for pages you want crawled and indexed, is there any benefit for not having the line at all?
Thanks!
-
Barry is spot on. You either have the meta robot line, or you do your robots.txt. I prefer to use the txt because it shows me all the rules I impose on the website at a glance and doesn't make me go and search each page individually if anything needs to be changed. Imagine having a website with 20000 pages where each is individually adjusted. If you suddenly need to change something it will take you forever. With a robots.txt it will literally take seconds.
-
Doesn't matter if the line's not there if you don't want the robots to do anything. Not including it just assumes the position that they can do whatever they're doing ie Index, Follow
All of those sites do have a few things in robots.txt though (and the meta is really for people that can't edit robots.txt)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Using same copy on different domain
I have a client that currently has a .com domain (not using hreflang) . They have a new partner in the UK and they want to replicate the website and use a .co.uk domain. It will be a different brand name. Will this cause any SEO issues?
Intermediate & Advanced SEO | | bedynamic0 -
Multiple H2 tags
Is it advisable to use only one H2 tag? The template designs for some reason is ended up with multiple H2 tags, I realise if any think it's that each one is that are important and it is all relative. Just trying to assess if it's worth the time and effort to rehash the template. Has anyone done any testing or got any experience? Thanks
Intermediate & Advanced SEO | | seoman101 -
How to optimize an ecommerce catalog that uses parameters only
Hello ! I am facing a problem concerning a client's website that has been developped using filters that create parameters - there are no categories. This means that, no matter what I choose as a filter, the page title, desc and my H1 stays the same. In a beautiful, unicorn rainbow filled world - I could just tell them to restructure their site with new categories/sub categories AND with filters. For SEO purposes and to find a temporary solution until we can change the architecture, what would be the best choice? Should we create individual pages that serves the same content as the catalog, but with rewritten URL, Title, Description and canonical?
Intermediate & Advanced SEO | | Charles-O
ie: http:/domain.com/catalog/?brand=moz canonical to http:/domain.com/catalog/brand/moz ? I noticed indeed.com does that (https://ca.indeed.com/SEO-Specialist-jobs vs https://ca.indeed.com/jobs?q=SEO+Specialist&l=) Should we dynamise the content depending on which filters has been selected? Of course, some filters are real filters that wouldn't attract or add any value (such as order by) Thanks for your input!0 -
Ecommerce SEO: Title Tags for pagination
Here's a specific question about title tags for ecommerce website... We've got lists of products (category list pages) that stretch for many pages... is there any benefit to added a something to make the title tag unique. For example: Page 1: <title></span><span class="html-tag" data-mce-mark="1">Category List Page Example</span><span class="html-tag" data-mce-mark="1"></title> Page 2: <title></span><span class="html-tag" data-mce-mark="1">Category List Page Example - Page 2</span><span class="html-tag" data-mce-mark="1"></title> Page 3: <title></span><span class="html-tag" data-mce-mark="1">Category List Page Example - Page 3</span><span class="html-tag" data-mce-mark="1"></title> FWIW, we've got the pagination and canonicalization nailed down tight. Moz crawl actual brought a dupe content issue based on title tags.
Intermediate & Advanced SEO | | 19prince0 -
Some Tools Not Recognizing Meta Tags
I am analyzing a site which has several thousands of pages, checking the headers, meta tags, and other on page factors. I noticed that the spider tool on SEO Book (http://tools.seobook.com/general/spider-test) does not seem to recognize the meta tags for various pages. However, using other tools including Moz, it seems the meta tags are being recognized. I wouldn't be as concerned with why a tool is not picking up the tags. But, the site suffered a large traffic loss and we're still trying to figure out what remaining issues need to be addressed. Also, many of those pages once ranked in Google and now cannot be found unless you do a site:// search. Is it possible that there is something blocking where various tools or crawlers can easily read them, but other tools cannot. This would seem very strange to me, but the above is what I've witnessed recently. Your suggestions and feedback are appreciated, especially as this site continues to battle Panda.
Intermediate & Advanced SEO | | ABK7170 -
Robots.txt Syntax
I have been having a hard time finding any decent information regarding the robots.txt syntax that has been written in the last few years and I just want to verify some things as a review for myself. I have many occasions where I need to block particular directories in the URL, parameters and parameter values. I just wanted to make sure that I am doing this in the most efficient ways possible and thought you guys could help. So let's say I want to block a particular directory called "this" and this would be an example URL: www.domain.com/folder1/folder2/this/file.html
Intermediate & Advanced SEO | | DRSearchEngOpt
or
www.domain.com/folder1/this/folder2/file.html In order for me to block any URL that contains this folder anywhere in the URL I would use: User-agent: *
Disallow: /this/ Now lets say I have a parameter "that" I want to block and sometimes it is the first parameter and sometimes it isn't when it shows up in the URL. Would it look like this? User-agent: *
Disallow: ?that=
Disallow: &that= What about if there is only one value I want to block for "that" and the value is "NotThisGuy": User-agent: *
Disallow: ?that=NotThisGuy
Disallow: &that=NotThisGuy My big questions here are what are the most efficient ways to block a particular parameter and block a particular parameter value. Is there a more efficient way to deal with ? and & for when the parameter and value are either first or later? Secondly is there a list somewhere that will tell me all of the syntax and meaning that can be used for a robots.txt file? Thanks!0 -
Wordpress Tag Pages - NoIndex?
Hi there. I am using Yoast Wordpress Plugin. I just wonder if any test have been done around the effects of Index vs Noindex for Tag Pages? ( like when tagging a word relevant to an article ) Thanks 🙂 Martin
Intermediate & Advanced SEO | | s_EOgi_Bear0 -
Canonical tag
Hi all, I have an ecommerce client and on the pages they have a drop down so customers can view via price, list etc. Natrurally I want a canonical tag on these pages, here's the question. as they have different pages of products, the canonical tag on http://www.thegreatgiftcompany.com/occassion/christmas#items-/occassion/christmas/page=7/?sort=price_asc,searchterm=,layout=grid,page=1 is to http://www.thegreatgiftcompany.com/occassion/christmas#items-/occassion/christmas/page=7. now, because the page=7 is a duplicate of the main page, shouldn't the canonical just be to the main page rather than page=7? Even when there is a canonical tag on the /Christmas/page=7 to the /Christmas page? hope that makes sense to everyone!
Intermediate & Advanced SEO | | KarlBantleman0