Robots file include sitemap
-
Hello,
I see that google, facebook and moz... have robots.txt include sitemap at the footer.
Eg: http://www.google.com.vn/robots.txtSitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Should I include my sitemap file (sitemap.xml) at the footer of robots.txt and why should do this?Thanks,
-
Fore sure. The reason is that not all sitemap files are simply sitemap.xml it may be sitemap.gzip sitemap.zip or in my case, sitmap_index.gzip. Also, some people may not be able to include their sitemap at root.
Including sitemap exacts in robots.txt gives a clear directive to each of the search engines exactly where to find your sitemap. Google and Bing/Yahoo will have no issue finding it as you probably submitted it to them but crawlers like ask.com will usually look at your robots.txt and skip your site if no map is placed in it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I include my company entity in meta description? (for ex. LLC. Inc.)
I'm re-writing my meta descriptions. I'm wondering if I should include the My Company, LLC. in the description or just My Company. Taking out the entity would save about 4 characters.
On-Page Optimization | | IcarusSEO1 -
Robots.txt Question for E-Commerce Sites
Hi All, I have a couple of e-commerce clients and have a question about URLs. When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout I'm not sure that I want these indexed. Could I be causing any harm/danger if I add this to the robots.txt file? /*? Any suggestions welcome! Gavin
On-Page Optimization | | IcanAgency0 -
Should I include unnecessary pages in the sitemap.xml
I have a lot of pages that I don't want Google to index, so for most of them, I used cannonical, were they were duplicates, noindex were I wanted to remove the pages, but the question is: Should I include these pages in the sitemap.xml, or just the important pages? Also should I include them in order to get the changes indexed fastet by Google?
On-Page Optimization | | Silviu0 -
SVG image files causing multiple title tags on page - SEO issue?
Does anyone have any experience with SVG image files and on-page SEO? A client is using them and it seems they use the title tag in the same way a regular image (JPG/PNG) would use an image ALT tag. I'm concerned that search engines will see the multiple title tags on the page and that this will cause SEO issues. Regular crawlers like Moz flag it as a second title tag, however it's outside the header and in a SVG wrap so the crawlers really should understand that this is a SVG title rather than a second page title. But is this the case? If anyone has experience with this, I'd love to hear about it.
On-Page Optimization | | mrdavidingram2 -
Yoast SEO sitemap link 404 problem
I have recently moved my wordpress blog from a subdomain into a directory e.g. www.mysite.com/blog/ and installed yeast SEO however when I go to the site map as directed in the pluign panel www.mysite.com/blog/sitemap_index.xml its not there are I get a 404 error? Any help much appreciated.
On-Page Optimization | | SamCUK0 -
Problemas with my htaccess file
Hi all, I have two doamins actived which content is the same. Regarding Google some of you guys told me that I wouldn't be penalized but If I wanted to do a redirection 301 it should be done from one domain to the domain which is my main market. And I just did it. But the problem is that although I have made the corresponding modifications in the htaccess file it doesn' work. When I write www.piensapiensa.com goes to piensapiensa.com (as I configured the WMT) and not to www.piensapiensa.es, in which my market is mainly present. Heres the code for the HTACCESS file: Options +FollowSymLinks +ExecCGI
On-Page Optimization | | juanmiguelcr
RewriteEngine on
RewriteCond %{HTTP_HOST} !^www.piensapiensa.es [NC]
RewriteRule (.*) http://www.piensapiensa.es/$1 [R=301,L]
<ifmodule mod_rewrite.c="">RewriteEngine On # uncomment the following line, if you are having trouble
# getting no_script_name to work
#RewriteBase / # we skip all files with .something
#RewriteCond %{REQUEST_URI} ..+$
#RewriteCond %{REQUEST_URI} !.html$
#RewriteRule .* - [L] # we check if the .html version is here (caching)
RewriteRule ^$ index.html [QSA]
RewriteRule ^([^.]+)$ $1.html [QSA]
RewriteCond %{REQUEST_FILENAME} !-f # no, so we redirect to our front web controller
RewriteRule ^(.*)$ index.php [QSA,L]</ifmodule> Thanks in advanced.0 -
Using Robots Meta Tag on Review Form Pages
I have gone over this so many times and I just can't seem to get it straight and hope someone can help me out with a couple of questions: Right now, on my dynamically created pages created by filters (located on the category pages) I am using rel""canonical" to point them to their respective category page. Should I also use the robots meta tag as well? Similarly, each product I have on my site has a review form on it and thus is getting indexed by Google. I have placed the same canonical tag on them as well pointing them to the page with the review form on it. In the past I used robots.txt to block google from the review pages but this didn't really do much. Should I be using the robots meta tag on these pages as well? If I used the robots meta tag should I noindex,nofollow? Thanks in advance, Jake
On-Page Optimization | | jake3720 -
Do I need a robots meta tag on the homepage of my site?
Is it recommended to include on the homepage of your site site? I would like Google to index and follow my site. I am using WordPress and noticed my homepage is not including this meta tag, therefore wondering if I should include it?
On-Page Optimization | | asc760