Robots file include sitemap
-
Hello,
I see that google, facebook and moz... have robots.txt include sitemap at the footer.
Eg: http://www.google.com.vn/robots.txtSitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Should I include my sitemap file (sitemap.xml) at the footer of robots.txt and why should do this?Thanks,
-
Fore sure. The reason is that not all sitemap files are simply sitemap.xml it may be sitemap.gzip sitemap.zip or in my case, sitmap_index.gzip. Also, some people may not be able to include their sitemap at root.
Including sitemap exacts in robots.txt gives a clear directive to each of the search engines exactly where to find your sitemap. Google and Bing/Yahoo will have no issue finding it as you probably submitted it to them but crawlers like ask.com will usually look at your robots.txt and skip your site if no map is placed in it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do you need to include the top menu on every single page of the site in the code?
When using cache: on google, and clicking on Text-only version, our site has the top menu gibberish on top? My feeling is that this take away SEO juice from our title and focus keyword. Our website is culinarydepotinc.com
On-Page Optimization | | Sammyh1 -
Which version of the homepage on the sitemap?
We have been wondering this for a while now. When we build our sitemaps, or when the yoast plugin does in WP we are often left with www.yourdomain.co.uk/ and www.yourdomain.co.uk/index.html in our sitemaps. Surely it isn't healthy to have both in the sitemap. Which one should we take out? Thanks
On-Page Optimization | | EveronSEO0 -
Wordpress sitemap url problem causing WMT errors
The following types of links are appearing in my webmaster tools crawl errors report under 'other'. I've noticed they are in my sitemaps ( I run wordpress and use a plugin called Google XML sitemaps). How do I get rid of this error? http://www.musicliveuk.com/bands/postname%/
On-Page Optimization | | SamCUK0 -
Should I include a meta description for my tags on posts
Hi Our crawl report detects that we have not put in a meta description for any of the tags we have used on our blog. (total tag words we have used is around 70). We have however, put a full description of each of our categories, do we really need to put a description for each of the tags we use? Will this really help us with SEO? Also can tags on posts be the same as the keyword your trying to rank the post/article about? For further information We are using Yoast.
On-Page Optimization | | lethalmarketing0 -
Is it redundant to include a redirect to my canonical domain (www) in my .htaccess file since I already have the correct rel="canonical" in my header?
I've been reading the benefits of each practice, but not found anyone mentioning whether it's really necessary to do both? Personally I try to stay clear of .htaccess rewrites unless it's absolutely necessary, since because I've read they can slow down a website.
On-Page Optimization | | HOPdigital0 -
How do i block an entire category/directory with robots.txt?
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now. The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc. I'm not really sure how i'd type it into the robots.txt file, or where to place the file. any help would be appreciated thanks
On-Page Optimization | | bricerhodes0 -
Best practice for Meta-Robots tag in categories and author pages?
For some of our site we use Wordpress, which we really like working with. The question I have is for the categories and authors pages (and similiar pages), i.e. the one looking: http://www.domain.com/authors/. Should you or should you not use follow, noindex for meta-robots? We have a lot of categories/tags/authors which generates a lot of pages. I'm a bit worried that google won't like this and leaning towards adding the follow, noindex. But the more I read about it, the more I see people disagree. What does the community of Seomoz think?
On-Page Optimization | | Lobtec0 -
Include the company/domain name in page titles and urls?
I know this isn't something that I would use site-wide but I'm wondering if it helps or hurts me to use my company name (also my domain name) in pages below the homepage. As an example, let's say I'm Home Depot. In the category pages off the homepage should I use Page names and urls like Home and Garden Supplies or Home and Garden Supplies at Home Depot? Or does it hurt me to reuse my company/domain name on multiple pages?
On-Page Optimization | | kdieruf0