Site Maps / Robots.txt etc
-
Hi everyone
I have setup a site map using a Wordpress pluggin: http://lockcity.co.uk/site-map/
Can you please tell me if this is sufficient for the search engines? I am trying to understand the difference between this and having a robots.txt - or do I need both?
Many thanks, Abi
-
Ahh I see thanks so much for your reply - is it best to get an XML site map rather than the one I have?
Thanks, Abi
-
A sitemap is for crawling your pages, either a user sitemap like you set up or a sitemap.xml file which is a streamlined way for Google to spider your site.
Robots.txt is used to direct the bots where NOT to go like if you don't want them crawling certain parts of your site like cgi-bin, js folder or images folder for example.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Stumped: Site No Longer Showing Up for Important Keywords
URL is: www.radianceofpalmbeach.comGreetings All:I have been working on our company's website for months, and I am finally at wit's end. The site was very out-of-date and had unfortunately been built upon some bad links before my arrival. My partner and I have redone the site with SEO best practices in mind: we created new content for the pages, and have been working diligently on correctly organizing the site. Despite everything we have done, our site has plummeted since September in terms of organic search. Here are some of my suspects: Panda/Penguin: a lot of the content of the old site had been copied. We did our best to make our content helpful and original, but I'm not sure we did enough. Also, many backlinks were suspect. I disavowed all that I didn't like Dec. 8. I have seen minor improvement, but not much. Name Change: Around late October, coinciding with one of the algorithm changes, the doctor insisted we change our name from New Radiance Med Spa to New Radiance Cosmetic Center. We noticed overnight tumbling, but it literally happened at the same time many were complaining about Penguin. Pages too far removed from root directory?: We tried to silo the site by category to make it specific, but I'm not sure if we went too far from the root directory. For example, our botox page is: http://www.radianceofpalmbeach.com/services/injectables/neuromodulators/botox-cosmetic/ -- Should it just be ./botox ? Everything is only one link away, so we didn't foresee a problem. No alternate forms of navigation: Our navigation is solely drop-down. Content Issues: Since the site launch, my boss has changed the organization of the site around. I don't think this should be a problem, but I honestly don't know. Technical Issues: We use a Wordpress site, and the designer has been pretty good about making the site clean and without errors, but perhaps there is something I am overlooking? ??: Despite these issues, I feel like our site should be considered better than many of our competitors who nonetheless perform much better than we do on important keyword searches. Type in "liposuction palm beach" or "botox palm beach" and we don't even come on page 1, whereas we used to dominate. Any suggestions would be greatly appreciated, as, like I said, we are stumped. I feel like I have looked up every possible problem, and with the above list, we feel frozen as to which direction to turn.Thanks in advance,Michael
On-Page Optimization | | mikedelseo0 -
What are the Best On-Site SEO Practices before an E-commerce Site Goes Live?
Hello, I’m working on a client’s E-commerce website. This website is not live yet. Before the site goes live, I am curious to know what the best practices of On-site SEO are. Please let me know from which factor should I start analyze? Thanks.
On-Page Optimization | | TopLeagueTechnologies0 -
Redirecting pages (old site to new site)
I have a question- there is one location, one set of pages for both the old and new site on the same host environment so when I did the redirect it get into a loop trying to redirect from itself to itself Not sure how its gonna affect SEO. Will pages get hit for duplicate content?
On-Page Optimization | | Yanez0 -
Should I use nofollow when interlinking large, networked sites?
My company runs a network of very large networked sites, each with thousands of content pages. In our main navigation we are currently not nofollowing links between these networked sites. The links appear on every single page in the top navigation, and there are thousands of pages on each site. I am worried this will look to Google like we have suspiciously received thousands of links from one domain - one link from every page on the domain. Should we be nofollowing these navigation links between the different sites in our network?
On-Page Optimization | | Natasha90040 -
Can we listed URL on Website sitemap page which are blocked by Robots.txt
Hi, I need your help here. I have a website, and few pages are created for country specific. (www.example.com/uk). I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page) I really appreciate your help. Thanks, Nilay
On-Page Optimization | | Internet-Marketing-Profs0 -
Advice needed about site
I would like to get advice about the site http://www.wb-3d.com regarding what we should do to get it highly on Google. This seems to be a tough one as there is not enough content on the site. I would immensly appeciate any suggestion to improve the rankings. Specifically, what we should tweet and update on facebook. Thanks
On-Page Optimization | | seoug_20050 -
How do we handle sitemaps in robots.txt when multiple domains point to same physical location?
we have www.mysite.net, www.mysite.se, www.mysite.fi and so on. all of these domains point to the same physical location on our webserver, and we replace texts given back to client depending on which domain he/she requested. My problem is this: How do i configure sitemaps in robots.txt when robots.txt is used by multiple domains? If I for instance put the rows Sitemap: http://www.mysite.net/sitemapNet.xml
On-Page Optimization | | nordicnetproducts
Sitemap: http://www.mysite.net/sitemapSe.xml in robots.txt, would that result in some cross submission error?0 -
Setting up a domain for a future site
Hi there, That may be a bit of a silly question to ask, but we've setup a new domain for an existing site. While the site is in the making, the site owners wants to already start promoting the new URL on stationeries etc. Hence, we need to setup the new URL so that it forwards to the site, but so that Google doesn't give it the history of a secondary (less important) domain. What is the best way to do this? Currently we've put in a 301 redirect, but will that bear no future consequences on the SEO of the site, when the site is moved to this new domain, and the old domain is 301 redirected. Thanks, SEOeclipse
On-Page Optimization | | Bozboz0