Using Site Maps Correctly
-
Hello
I'm looking to submit a sitemap for a post driven site with over 5000 pages.
The site hasn't got a sitemap but it is indexed by google - will submitting a sitemap make a difference at this stage?
Also, most free sitemap tools only go up to 5000 pages, and I'm thinking I would try a sitemap using a free version of the tool before I buy one - If my site is 5500 pages but I only submit a sitemap for 5000 (I have no control of which pages get included in the sitemap) would this have a negative effect for the pages that didn't get included?
Thanks
-
Submitting a sitemap in Webmaster Console is always a good idea at any stage. If your website URLs are crawled and indexed in search engines than there will be no negative impact of it but in the longer run if you add more pages sitemap will defiantly a help.
If you are using CMS like WordPress, Joomla, Zencart or any other they all have extensions and plugins in their directory that will help you generate the sitemap of your current site and will add links as soon as you will add more pages.
Rest peter explains almost everything in detail like if you have URL issues and issues with crawling and indexing.
If you have a custom CMS, I think you should seriously consider the idea by Peter as this is something you need on regular basis anyways!
Hope this helps!
-
It's hard to tell without seeing your URL architecture.
First there are two specific terms and you never, never ever should forget them. They are - crawling and indexing. Once you prepare sitemap and submit there (or include in robots.txt) all bots get some map of your site and start crawling pages based on their crawling budget for your site. In crawling process they MAY find new pages that doesn't include in this map and will crawl them too. Again this is based on your crawling budget.
So when you submit sitemap - bot will get within seconds list of "non-crawled" 5000 pages and will start crawl them. Then he can find missed 500 pages and will crawl them too. Tricky is that when you update sitemap - he can detect quick changes there and start recrawling them again. But for missed 500 pages he can visit you again to check them for changes. And this will be also under your crawling budget. But if pages there isn't changed often - isn't big deal.
So you shouldn't hesitated about negative impact there. Only negative impact can happen if you have some serious URL architecture issues and messy URLs there. Then submitting partial sitemap can obfuscate this issues and some of your URLs to remain non-crawled.
Technically in SearchConsole you can see sitemap statistics like submitted and indexed. In perfect world numbers should be almost equal with little difference. But if you see huge difference between them - then you're in trouble. For example - on some site i have sitemap with submitted 44,950 pages and indexed of them was 29,643. This is pure example site crawling troubles or sitemap troubles. Because 1/3 of all pages isn't indexed at all.
PS: I forgot. You should use own CMS plugin for generating sitemap inside. Even if your CMS was custom made you should write (or hire someone) to create plugin inside. It's near 20-30 lines of write-here-your-favorite-language (PHP/Python/Perl/Ruby) and isn't big deal. This plugin will minimize crawling time from 3rd party sitemap generator tool because CMS already have all information inside and just need to be exported to XML.
-
It would definitely be better to submit a complete sitemap. If your site is built in Wordpress, Joomla, Magento, or many other standard CMS, it should have the ability to generate a full sitemap. Plugins like Yoast or Google Sitemaps help. Just depends on the site.
Otherwise you can probably get any pro SEO or agency to create a full 5500+ sitemap for you for $100 bucks or so. PM me if you need more help.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Tracking using Custom URLs - is this viable?
Hi Moz community! I’ll try to make this question as easy to understand as possible, but please excuse me if it isn’t clear. Just joined a new team a few months ago and found out that on some of our most popular pages we use “custom URLs” to track page metrics within Google Analytics. NOTE: I say “custom URLs” because that is the best way for me to describe them. As an example: This page exists to our users: http://usnews.rankingsandreviews.com/cars-trucks/Ram_HD/2012/photos-interior/ But this is the URL we have coded on the page: cars-trucks/used-cars/reviews/2012-Ram-HD/photos-interior/ (within the custom variance script labeled as “var l_tracker=” ) It is this custom URL that we use within GA to look up metrics about this page. This is just one example of many across our site setup to do the same thing Here is a second example: Available page to user: http://usnews.rankingsandreviews.com/cars-trucks/Cadillac_ATS/2015/ Custom “var l_tracker=” /cars-trucks/2015-Cadillac-ATS/overview/ NOTE: There is a small amount of fear that the above method was implemented years ago as a work-around to a poorly structured URL architecture. Not validated, but that is a question that arose. Main Questions: Is the above implementation a normal and often used method to track pages in GA? (coming from an Omniture company before – this would not be how we handled page level tracking) Team members at my current company are divided on this method. Some believe this is not a proper implementation and are concerned that trying to hide these from Google will raise red flags (i.e. fake URLs in general = bad) I cannot find any reference to this method anywhere on the InterWebs - If method is not normal: Any recommendations on a solution to address this? Potential Problems? GA is currently cataloguing these tracking URLs in the Crawl Error report. Any concerns about this? The team wants to hide the URLs in the Robots.txt file, but some team members are concerned this may raise a red flag with Google and hurt us more than help us. Thank you in advance for any insight and/or advice. Chris
Reporting & Analytics | | usnseomoz0 -
Using Canonicals on category or link pages
Hey, If a site has category or tag pages showing in search results for a particular keyword - sometimes higher than the page you would like to rank - could using canonical redirects on categories and tags be a solution? Thanks
Reporting & Analytics | | wearehappymedia0 -
Using Google Analytics network service provider
Hi What are people's views on using the Technology > Network > Service provider part of Google Analytics as a way to find companies that have visited a website? Then potentially contact them companies etc? Thanks
Reporting & Analytics | | Hughescov0 -
Having Issue with Site Search in Analytics
Hi Mozzers, We launched a website in October 2012 and have added in the settings(Google analytics) of that profile "Do Track Site Search" since we have a search box on the website. The site search report worked for 10 days and it was over(from end of december till beginning of January 2013). Since then I have been trying to understand this issue. I have added all the query search terms possible, but still not showing any signs of life. At this point I am not sure what to do? Some Help would be appreciated! Search URL= subdomain.example.com**/search/node/**.... Thanks! z93cGUZ.png
Reporting & Analytics | | Ideas-Money-Art0 -
Using advance segments or primary dimensions?
Hi guys, I haven't played around with advance segments enough to know the answer to this, so looking for some help. What I want to track is a keyword through the whole analytics, so I created an advance segment for "KEYWORD". This segment is Include - Keyword - "KEYWORD". Now when I remove the advance segment and do the following: Traffic Sources - Organic - Primary Dimension - Keyword - "KEYWORD" The amounts of visits differ a lot. I'm guessing this has to do with sessions? Looking for a clarification on this.
Reporting & Analytics | | William.Lau0 -
Use Google Spreadsheet for Top Keywords Report
I've been using the super-awesome Google doc described here http://www.seomoz.org/blog/custom-reporting-using-google-analytics-and-google-docs for all kinds of reporting and I've customized it quite a bit. But I can't figure out how to feed text-based fields from GA into this doc. For example, can I make it so this doc spits out top 10 keywords? Thanks!
Reporting & Analytics | | Oxfam_America0 -
Setting Up Google Analytic with Sub Folder Sites
What is the best way of setting up Google Analytic for a website that has many sub folders? The main site is example.com and it has 40 sub folder sites like example.com/uk example.com/France etc etc Would it be advised to track a single domain in Google Analytic then create filters for the sub folder sites. Filters > Include traffic from > Sub directories Also with this method is it possible to view overall incoming website stats for everything? Previous experience would be great with this thanks 🙂
Reporting & Analytics | | daracreative0 -
Should we add the city to our keywords for a site that is only local?
This is one of those things I have done for a long time and all of a sudden asked myself was it necessary: For our local clients, we add the city name (Houston, KC, Birmingham) after each keyword. An example would be TestSite.com/big-tester-houston A Title Tag might be Big Tester Houston | Test Site, etc. Where appropriate we do the same with H1 or H2's and occasionally in the content we will use the city name. The thought being that since the site is only for a given city, it will be deemed more relevant than a site from outside.( I understand there are other factors in SEO; this is a specific question around adding the city). Yes, we also optimize with local directories/citation sites. Is this overkill, is it even worthwhile? Is there any evidence one way or another? I would love some strong opinions backed up with something other than anecdotal evidence where possible.
Reporting & Analytics | | RobertFisher0