When I try creating a sitemap, it doesnt crawl my entire site.

[TheSquareFoot]

We just launched a new Ruby app at (used to be a wordpress blog) -

http://www.thesquarefoot.com

We have not had time to create an auto-generated sitemap, so I went to a few different websites with free sitemap generation tools. Most of them index up to 100 or 500 URLS. Our site has over 1,000 individual listings and 3 landing pages, so when I put our URL into a sitemap creator, it should be finding all of these pages. However, that is not happening, only 4 pages seem to be seen by the crawlers.

TheSquareFoothttp://www.thesquarefoot.com/http://www.thesquarefoot.com/users/sign_inhttp://www.thesquarefoot.com/searchhttp://www.thesquarefoot.com/renters/sign_upThis worries me that when Google comes to crawl our site, these are the only pages it will see as well.  Our robots.txt is blank, so there should be nothing stopping the crawlers from going through the entire site. Here is an example of one of the 1,000s of pages not being crawled****http://www.thesquarefoot.com/listings/Houston/TX/77098/Central_Houston/3910_Kirby_Dr/Suite_204Any help would be much appreciated!

[TheSquareFoot]

Thanks for you help, can I ask one more question -

We just submitted a new sitemap to google for our new rails app -

http://www.thesquarefoot.com/sitemap.xml

Which has over 1,300 pages, however Google is only seeing 114. About 1,025 are in the listings folder / 250 blog posts / and 15 landing pages.

Any help would be appreciated!

Aron

sitemap.png sitemap.png

[matbennett]

I'd worry less about the sitemaps and more about internal linking structure.  The problem you are having with crawlers is as symptom of the linking problem.

Most of your content seems to be on the other side of a search form.  When crawlers, including those from search engines, explore you site they are looking for href links to follow - they will not submit forms.

If then you want the other content to be indexed then you need to provide a crawl path to it.  Could you add links to each neighbourhood on page somewhere so that there is path to follow?  That might lead on to further questions about your url structure and use of ajax too.

The general principal is that you should link to content you want to rank. Many will argue that a sitemap removes that necessity, but links provide more information that a list of URLs and I certainly wouldn't rely on sitemaps alone to get content indexed let alone ranked.