Can't generate a sitemap with all my pages
-
I am trying to generate a site map for my site nationalcurrencyvalues.com but all the tools I have tried don't get all my 70000 html pages... I have found that the one at check-domains.com crawls all my pages but when it writes the xml file most of them are gone... seemingly randomly.
I have used this same site before and it worked without a problem. Can anyone help me understand why this is or point me to a utility that will map all of the pages?
Kindly,
Greg
-
Thank you all for the responses... I found them all helpful. I will look into creating my own sitemap with the IIS tool.
I can't help the 70k pages but the URLS are totally static. I guess I can make a site map for all the aspx pages and then other one for all the lowest level .html pages.
Thanks everyone!
-
I definitely agree with Logan. The max for an XML sitemap for Search Console is 50,000 URLs, so you won't be able to fit all of yours into one.
That being the case, divide them into different sitemaps by category or type, then list all of those in one directory sitemap and submit that. Now you can see indexation by page type on your website.
Finally, I have to ask why you are doing this with a third party tool and creating a static sitemap as opposed to creating a dynamic one that can update automatically when you publish new content? If your site is static and you're not creating new pages, then your approach might be ok, but otherwise I'd recommend investigating how you build a dynamic XML sitemap that updates with new content.
Cheers!
-
Looking at your site how sure are you that you need 70,000 pages?
For the sitemap I would stop trying to use a website and do it yourself. It looks like you are running IIS. They have a sitemap generator that you can install on a server easily and run it there. It looks like you have GoDaddy, they catch a lot of crap but I have always found their technical support to be top notch. If you can't figure out how to do it on the server I would give them a call.
-
Greg,
Have you tried creating multiple XML sitemaps by section of the site, like by folder or by product detail pages? 70,000 is a huge amount of URLs and even if you could get them all on one sitemap, I wouldn't recommend it. Nesting sitemaps into an index sitemap can help Google understand your site structure and make it easier for you to troubleshoot indexing problems should they arise.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google doesn't show proper meta for my subpage, how to fix it?
We have a subdomain blog.companyname.com. I am working on its English version blog.companyname.com/en but for some reason Google shows meta description from blog.companyname.com in search results which is not in Englsih language. How do I force google to show blog.companyname.com/en 's own meta?
Intermediate & Advanced SEO | | SofyaFr0 -
International SEO: can I choose only certain pages for subfolder?
For a client we are discussing international SEO options. I have pushed against CCTLD because they do not have the resources to manage multiple sites. Instead we want to go the subfolder route: .com/uk/ My question is whether we can properly create a subfolder version that only includes a handful of pages rather than the whole site - so 5-10 pages vs 4k. Is that possible? I'd love your thoughts. International SEO is not my strong suit. Also - if the subfolder for .com/uk/ content is almost entirely the same as the us-based .com is that a problem? Thanks!
Intermediate & Advanced SEO | | JBMediaGroup0 -
Taken a canonical off a page to let it rank with new unique content - what more can I do?
A week ago, I took a canonical off of a page that was pointing to the homepage for a very big, generic search term for my brand as we felt that it could have been harming our rankings (as it wasn't a true canonical page). A week in and our rankings for the term have dropped 7 positions out of page 1 and the page we want to rank instead is nowhere to be seen. Do I hang fire? As such a big search term, it's affecting traffic, but I don't want to make any rash decisions. Here's a bit more info: For arguments sake, let's call the search term we're going after 'Boots', with the URL where the canonical was placed of /boots. The canonical went to the root domain as we sell, well... boots. At the time, the homepage was ranking for Boots on page 1 and we wanted to change this so that the Boots page ranked for that term... all logical right? We did the following: Took off mentions of Boots from meta on the homepage and made sure it was optimised for on the boots page. Took the canonical off of /boots. Used GSC to fetch & ask Google to recrawl "/boots". Resubmitted the sitemap. Do I hang fire on running back to the safety of ranking for boots on the homepage? Do I risk keyword cannibalisation by adding the search terms back to the homepage?
Intermediate & Advanced SEO | | Kelly_Edwards0 -
How does google treat dynamically generated content on a page?
I'm trying to find information on how google treats dynamically generated content within a webpage? (not dynamic urls) For example I have a list of our top 10 products with short product descriptions and links on our homepage to flow some of the pagerank to those individual product pages. My developer wants to make these top products dynamic to where they switch around daily. Won't this negatively affect my seo and ability to rank for those keywords if they keep switching around or would this help since the content would be updated so frequently?
Intermediate & Advanced SEO | | ntsupply0 -
Pull meta descriptions from a website that isn't live anymore
Hi all, we moved a website over to Wordpress 2 months ago. It was using .cfm before, so all of the URLs have changed. We implemented 301 redirects for each page, but we weren't able to copy over any of the meta descriptions. We have an export file which has all of the old web pages. Is there a tool that would allow us to upload the old pages and extract the meta descriptions so that we can get them onto the new website? We use the Yoast SEO plugin which has a bulk meta descriptions editor, so I'm assuming that the easiest/most effective way would be to find a tool that generates some sort of .csv or excel file that we can just copy and paste? Any feedback/suggestions would be awesome, thanks!
Intermediate & Advanced SEO | | georgetsn0 -
Rankings drop - we've added user reviews, are they causing over optimisation on page?
Hello Hopefully can get a few opinions on this. We've added some user reviews to our website for key products. We added these approximately 3-4 weeks ago. In the last week we've seen keyword rankings drop on the pages they've been added to. For example see: http://www.naturalworldsafaris.com/wildlife/primates.aspx This page ranked well for both gorilla safari and gorilla safaris but both terms have dropped considerably (12 to 20 checking Google UK on the Moz rank checker). Due to the formatting required for the Rich Snippets (and we have the user review stars in the SERPS) the term "Gorilla safari" is perhaps becoming a bit spammy on the page. Another example would be "Borneo holidays" (up and down in the SERPS between 12-18) on this page: http://www.naturalworldsafaris.com/destinations/far-east/borneo.aspx Do you feel that these fluctuations in keyword ranking could be to do with this? Thanks
Intermediate & Advanced SEO | | KateWaite0 -
Can submitting sitemap to Google webmaster improve SEO?
Can creating fresh sitemap and submitting to Google webmaster improve SEO?
Intermediate & Advanced SEO | | chanel270 -
NOINDEX listing pages: Page 2, Page 3... etc?
Would it be beneficial to NOINDEX category listing pages except for the first page. For example on this site: http://flyawaysimulation.com/downloads/101/fsx-missions/ Has lots of pages such as Page 2, Page 3, Page 4... etc: http://www.google.com/search?q=site%3Aflyawaysimulation.com+fsx+missions Would there be any SEO benefit of NOINDEX on these pages? Of course, FOLLOW is default, so links would still be followed and juice applied. Your thoughts and suggestions are much appreciated.
Intermediate & Advanced SEO | | Peter2640