Robots file include sitemap
-
Hello,
I see that google, facebook and moz... have robots.txt include sitemap at the footer.
Eg: http://www.google.com.vn/robots.txtSitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Should I include my sitemap file (sitemap.xml) at the footer of robots.txt and why should do this?Thanks,
-
Fore sure. The reason is that not all sitemap files are simply sitemap.xml it may be sitemap.gzip sitemap.zip or in my case, sitmap_index.gzip. Also, some people may not be able to include their sitemap at root.
Including sitemap exacts in robots.txt gives a clear directive to each of the search engines exactly where to find your sitemap. Google and Bing/Yahoo will have no issue finding it as you probably submitted it to them but crawlers like ask.com will usually look at your robots.txt and skip your site if no map is placed in it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I include my company entity in meta description? (for ex. LLC. Inc.)
I'm re-writing my meta descriptions. I'm wondering if I should include the My Company, LLC. in the description or just My Company. Taking out the entity would save about 4 characters.
On-Page Optimization | | IcarusSEO1 -
Two Robots.txt files
Hi there Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one. Thanks in advance! Cheers,
On-Page Optimization | | TrulyTravel
Satla0 -
SVG image files causing multiple title tags on page - SEO issue?
Does anyone have any experience with SVG image files and on-page SEO? A client is using them and it seems they use the title tag in the same way a regular image (JPG/PNG) would use an image ALT tag. I'm concerned that search engines will see the multiple title tags on the page and that this will cause SEO issues. Regular crawlers like Moz flag it as a second title tag, however it's outside the header and in a SVG wrap so the crawlers really should understand that this is a SVG title rather than a second page title. But is this the case? If anyone has experience with this, I'd love to hear about it.
On-Page Optimization | | mrdavidingram2 -
Noindex child pages (whose content is included on parent pages)?
I'm sorry if there have been questions close to this before... I've using WordPress less like a blogging platform and more like a CMS for years now... For content management purposes we organize a lot of content around Parent/Child page (and custom-post-type) relationships; the Child pages are included as tabbed content on the Parent page. Should I be noindexing these child pages, since their content is already on the site, in full, on their Parent pages (ie. duplicate content)? Or does it not matter, since the crawlers may not go to all of the tabbed content? None of the pages have shown up in Moz's "High Priority Issues" as duplicate content but it still seems like I'm making the Parent pages suffer needlessly... Anything obvious I'm not taking into consideration? By the by, this is my first post here @ Moz, which I'm loving; this site and the forums are such a great resource! Anyways, thanks in advance!
On-Page Optimization | | rsigg0 -
Problemas with my htaccess file
Hi all, I have two doamins actived which content is the same. Regarding Google some of you guys told me that I wouldn't be penalized but If I wanted to do a redirection 301 it should be done from one domain to the domain which is my main market. And I just did it. But the problem is that although I have made the corresponding modifications in the htaccess file it doesn' work. When I write www.piensapiensa.com goes to piensapiensa.com (as I configured the WMT) and not to www.piensapiensa.es, in which my market is mainly present. Heres the code for the HTACCESS file: Options +FollowSymLinks +ExecCGI
On-Page Optimization | | juanmiguelcr
RewriteEngine on
RewriteCond %{HTTP_HOST} !^www.piensapiensa.es [NC]
RewriteRule (.*) http://www.piensapiensa.es/$1 [R=301,L]
<ifmodule mod_rewrite.c="">RewriteEngine On # uncomment the following line, if you are having trouble
# getting no_script_name to work
#RewriteBase / # we skip all files with .something
#RewriteCond %{REQUEST_URI} ..+$
#RewriteCond %{REQUEST_URI} !.html$
#RewriteRule .* - [L] # we check if the .html version is here (caching)
RewriteRule ^$ index.html [QSA]
RewriteRule ^([^.]+)$ $1.html [QSA]
RewriteCond %{REQUEST_FILENAME} !-f # no, so we redirect to our front web controller
RewriteRule ^(.*)$ index.php [QSA,L]</ifmodule> Thanks in advanced.0 -
Can we listed URL on Website sitemap page which are blocked by Robots.txt
Hi, I need your help here. I have a website, and few pages are created for country specific. (www.example.com/uk). I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page) I really appreciate your help. Thanks, Nilay
On-Page Optimization | | Internet-Marketing-Profs0 -
Can duplicate content issues be solved with a noindex robot metatag?
Hi all I have a number of duplicate content issues arising from a recent crawl diagnostics report. Would using a robots meta tag (like below) on the pages I don't necessarily mind not being indexed be an effective way to solve the problem? Thanks for any / all replies
On-Page Optimization | | joeprice0 -
Google Sitemap
Does adding a Google Sitemap to webmaster tools REALLY help SEO? If so, are there any resources for help creating one? Here is my site: http://www.petmedicalcenter.com Thanks,
On-Page Optimization | | PMC-312087
Brant0