Site Maps / Robots.txt etc
-
Hi everyone
I have setup a site map using a Wordpress pluggin: http://lockcity.co.uk/site-map/
Can you please tell me if this is sufficient for the search engines? I am trying to understand the difference between this and having a robots.txt - or do I need both?
Many thanks, Abi
-
Ahh I see thanks so much for your reply - is it best to get an XML site map rather than the one I have?
Thanks, Abi
-
A sitemap is for crawling your pages, either a user sitemap like you set up or a sitemap.xml file which is a streamlined way for Google to spider your site.
Robots.txt is used to direct the bots where NOT to go like if you don't want them crawling certain parts of your site like cgi-bin, js folder or images folder for example.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Meta Robots information
Hi, I have a question about the Meta Robots information Accoarding to the Moz bar, our page uses the meta robots noodp and noydir. Our competitor uses
On-Page Optimization | | AdoBike
INDEX,FOLLOW I read that noodp and noydir are dated and not used anymore. Is it wise to use INDEX FOLLOW instead for better SEO? Thanks in advance!1 -
Header/Menu Links
Hey there, I seem to have heard mixed results on this. We have a LOT of pages that we are trying to tell google are important. Does it make that big of a difference if those pages are linked in the homepage/header menu? Or mega menu for example? We are essentially a review site, so we potentially have thousands of links that could be implemented into the menu. Thoughts are greatly appreciated!
On-Page Optimization | | HashtagHustler0 -
How/when do you trim your article pages?
Hi all, I wanted to ask what have you all done to your article section that has made a difference to the site's SEO performance? What's the criteria for deciding which articles top cut or 301 redirect to the main page For example: 1. you have a slightly irrelevant article to your website topic, low conversions, but a lot of quality links to page, but no primary keywords in article title. 2. You have a slightly irrelevant article to your website, low conversion, but a lot of quality links AND you have your site's primary keywords in article title 3. You have a relevant topic to the site, but low engagement and low conversions Thanks for all your help!
On-Page Optimization | | SDavis111 -
Random /feed 404 error from a wordpress site
My Moz Analytics report shows a 404 error on a page which I think should not exist at all. The URL is http://henryplumbingco.com/portfolio-item/butler-elementary/feed/. When I checked webmaster tools, it looks like there are a number of random /feed urls throwing 404 errors. I am using WordPress and the Enfold theme. Anyone know how to get rid of these errors? Thanks,
On-Page Optimization | | aj6130 -
HTML Site SEO (NO CMS)
I have got a client site, which is dated (2007) and has not been shifted to any recognised CMS yet. It is HTML based. Is it possible to SEO on such a site? Is it even worth it? If it is possible to do SEO on this, any suggestions will be highly appreciated. Thank you.
On-Page Optimization | | ArthurRadtke3 -
Duplicate content with a trailing slash /
Hi, I 've pages like this: A) www.example.com/file/ B) www.example.com/file Just two questions: Does Google see this as duplicate content? Best to 301 redirect B to A? Many thanks Richard PS I read previous threads re the subject, it sounded like there was a bug in SEOMoz but I was not absolutely clear. Apologies if this is going over old ground.
On-Page Optimization | | Richard5550 -
Site Downtime - Will pages be reindexed?
I recently had site downtime of about 10 days for a site I'm working on which is about 6 months old. There were naturally many page not found errors in webmaster tools as google had tried to crawl the site during the downtime but I've noticed that google has now dropped some of the pages. Are they likely to be reindexed? Google has crawled some of my pages today but as yet the ones that have been dropped from the index haven't reappeared. (the site has been live again for a week). Should they reappear when the pages are next crawled?
On-Page Optimization | | SamCUK0 -
Should our social network put all of our member profiles in the site map?
We're working on some changes to our site and want to add a site map file to give some hints to search engines on our member profiles. Our site has millions of member profiles. I want to ensure that Google indexes them. Is there a benefit to submitting millions of profiles across multiple site maps? Is it better to pick a sample of those and submit them? Most member profiles have links to other member profiles. Is there any benefit to taking a single site map of 50k links and then changing it at some frequency so that all of the member profiles are eventually indexed? Thanks.
On-Page Optimization | | JoeCotellese810