Not using a robot command meta tag
-
Hi SEOmoz peeps.
Was doing some research on robot commands and found a couple major sites that are not using them.
If you check out the code for these:
http://www.zappos.com/product/7787787/color/92100
You fill not find a meta robot command line.
Of course you need the line for any noindex, nofollow, noarchive pages.
However for pages you want crawled and indexed, is there any benefit for not having the line at all?
Thanks!
-
Barry is spot on. You either have the meta robot line, or you do your robots.txt. I prefer to use the txt because it shows me all the rules I impose on the website at a glance and doesn't make me go and search each page individually if anything needs to be changed. Imagine having a website with 20000 pages where each is individually adjusted. If you suddenly need to change something it will take you forever. With a robots.txt it will literally take seconds.
-
Doesn't matter if the line's not there if you don't want the robots to do anything. Not including it just assumes the position that they can do whatever they're doing ie Index, Follow
All of those sites do have a few things in robots.txt though (and the meta is really for people that can't edit robots.txt)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Google ignore duplicate meta descriptions?
Hi there SEO mozzers, I am dealing with a website that has duplicate meta descriptions (we know is bad).As a punishment, Google totally ignores the meta descriptions and picks content from the website and displays it in SERP. I already read the https://moz.com/blog/why-wont-google-use-my-meta-description but I was wondering if there is more information/knowledge out there. Any tips are appreciated!
Intermediate & Advanced SEO | | Europarl_SEO_Team0 -
Using hreflang for international pages - is this how you do it?
My client is trying to achieve a global presence in select countries, and then track traffic from their international pages in Google Analytics. The content for the international pages is pretty much the same as for USA pages, but the form and a few other details are different due to how product licensing has to be set up. I don’t want to risk losing ranking for existing USA pages due to issues like duplicate content etc. What is the best way to approach this? This is my first foray into this and I’ve been scanning the MOZ topics but a number of the conversations are going over my head,so suggestions will need to be pretty simple 🙂 Is it a case of adding hreflang code to each page and creating different URLs for tracking. For example:
Intermediate & Advanced SEO | | Caro-O
URL for USA: https://company.com/en-US/products/product-name/
URL for Canada: https://company.com/en-ca/products/product-name /
URL for German Language Content: https://company.com/de/products/product-name /
URL for rest of the world: https://company.com/en/products/product-name /1 -
Default Robots.txt in WordPress - Should i change it??
I have a WordPress site as using theme Genesis i am using default robots.txt. that has a line Allow: /wp-admin/admin-ajax.php, is it okay or any problem. Should i change it?
Intermediate & Advanced SEO | | rootwaysinc0 -
Panda Updates - robots.txt or noindex?
Hi, I have a site that I believe has been impacted by the recent Panda updates. Assuming that Google has crawled and indexed several thousand pages that are essentially the same and the site has now passed the threshold to be picked out by the Panda update, what is the best way to proceed? Is it enough to block the pages from being crawled in the future using robots.txt, or would I need to remove the pages from the index using the meta noindex tag? Of course if I block the URLs with robots.txt then Googlebot won't be able to access the page in order to see the noindex tag. Anyone have and previous experiences of doing something similar? Thanks very much.
Intermediate & Advanced SEO | | ianmcintosh0 -
Should I Use City Name in URL?
Having a website designed for a car dealership and deciding what attributes to use in the URL. Should I include the city name in the URL? Or does that help for SEO purposes? Other ideas of what to research or try are appreciated too. Thanks 🙂
Intermediate & Advanced SEO | | kylesuss0 -
Help with canonical tag
hello- i got this recommendation <dl> <dt>Recommendation</dt> <dd>Add a canonical URL tag referencing this URL to the header of the page</dd> <dd>from my "report card" and i see also that i have a lot of issues with duplicate content but i really dont have any duplicate content on my site.</dd> <dd>the crawl has apparently marked every post in my blog as duplicate page content.</dd> <dd>and the "use canonical tag" suggestion keeps appearing as a fix to my problems.</dd> <dd>could you please help me with ------How do i create a canonical tag?</dd> <dd>is it just rel=canonical?</dd> <dd>and where do i put it?</dd> <dd>i should put it on every page right?</dd> <dd>or with CSS my webmaster could probably do it very quickly right?</dd> <dd>i get the basic concept behind rel=canonical but i cant say i fully understand it -</dd> <dd>i need some help with regard to how and where this tag should be placed.</dd> <dd>thanks,</dd> <dd>erik
Intermediate & Advanced SEO | | Ezpro9
</dd> <dd>.</dd> </dl>0 -
Blocking Dynamic URLs with Robots.txt
Background: My e-commerce site uses a lot of layered navigation and sorting links. While this is great for users, it ends up in a lot of URL variations of the same page being crawled by Google. For example, a standard category page: www.mysite.com/widgets.html ...which uses a "Price" layered navigation sidebar to filter products based on price also produces the following URLs which link to the same page: http://www.mysite.com/widgets.html?price=1%2C250 http://www.mysite.com/widgets.html?price=2%2C250 http://www.mysite.com/widgets.html?price=3%2C250 As there are literally thousands of these URL variations being indexed, so I'd like to use Robots.txt to disallow these variations. Question: Is this a wise thing to do? Or does Google take into account layered navigation links by default, and I don't need to worry. To implement, I was going to do the following in Robots.txt: User-agent: * Disallow: /*? Disallow: /*= ....which would prevent any dynamic URL with a '?" or '=' from being indexed. Is there a better way to do this, or is this a good solution? Thank you!
Intermediate & Advanced SEO | | AndrewY1 -
What <h>Tag to Use For Global Navigation</h>
I've read several blogs discussing how including more than one H1 per page is a serious no no. However, what is the most effective <h>tag to use for your global navigation system. Or should it not be an <h>tag period?</h></h>
Intermediate & Advanced SEO | | calin_daniel0