Google Indexing of Site Map
-
We recently launched a new site - on June 4th we submitted our site map to google and almost instantly had all 25,000 URL's crawled (yay!).
On June 18th, we made some updates to the title & description tags for the majority of pages on our site and added new content to our home page so we submitted a new sitemap.
So far the results have been underwhelming and google has indexed a very low number of the updated pages. As a result, only a handful of the new titles and descriptions are showing up on the SERP pages.
Any ideas as to why this might be? What are the tricks to having google re-index all of the URLs in a sitemap?
-
No problem, its actually really easy:
https://www.google.com/webmasters/tools/googlebot-fetch
Once you have selected your account, add the URL and then submit to index. I would do the homepage first and for that page, use the "Crawl this URL and its direct links" option. Then for the subpages do the "Crawl only this URL" option. It can also help to do the "Crawl this URL and its direct links" for any of your top level menu items to help speed things up.
"For example, i just checked a page and saw that some images weren't being indexed." Does your robots file allow specific access to those pages? If not, here is how you can set it to do so. This will also allow Google's partners to access your images. Add this to the bottom of your robots file:
User-agent: Googlebot-Image
Allow: /images/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /
User-agent: Mediapartners-Google*
Allow: /
Sitemap: http://www.YOURSITEHERE.com/sitemap.xml -
Thank you!! I'll take a look through the google resource. Also the site:domain search reviled 35,000 results.
The results are there, just not reindexed.
-
David,
Thanks for your response. This is exactly what we've seen with the initial spike in ranking and now with things settling down. I'll make sure the team has the crawl requests to daily (which I think it is).
For fetch as google - what's the best way that you've used this? For example, i just checked a page and saw that some images weren't being indexed. If I correct the issue, can I just use "Submit to Index"?
Thanks!!!!
-
In the 1000's of sites we have submitted, all show an initial spike in ranking and indexing before things settle down for the long haul. It seems like Google does a "best guess" scenario, before they take the time to fully crawl and analyze all of the URL's and rank them accordingly. As always, resubmit the pages through all webmaster tools (Bing too!) so that they are always aware of the most recent updates. If you are planning on updating the pages frequently, I would edit your crawl request to daily in your sitemap. They probably won't do it anyway, but you can try
Use the fetch as Google religiously when you update. It is your friend
-
Hi there
Did you read through Google's indexing resources?
I would also try doing a quick "site:yourdomain.com" and see how many pages Google pulls up - that's a more accurate representation of what's indexed from your site. This is reflected in the resource above:
"Sometimes the data we show in Index Status is not fully reflected in Google Search results." I suggest reading through the resource and also performing that search. Google indexing your sitemap is a waiting game, you're on the watch, just be patient!
Hope this helps! Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why isn't our new site being indexed?
We built a new website for a client recently. Site: https://www.woofadvisor.com/ It's been live for three weeks. Robots.txt isn't blocking Googlebot or anything. Submitted a sitemap.xml through Webmasters but we still aren't being indexed. Anyone have any ideas?
Technical SEO | | RobbieD910 -
Google is not indexing my new URL structure. Why not?
Hi all, We launched a new website for a customer on April 29th. That same day we resubmitted the new sitemap & asked Google to fetch the new website. Screenshot is attached of this (GWT Indexed). However, when I look at Google Index (see attachment - Google Index), Automated Production's old website URL's still appear. It's been two weeks. Is it normal for Google's index to take this long to update? Thanks for your help. Cole VoLPjhy vfxVUsO
Technical SEO | | ColeLusby0 -
How can I tell Google not to index a portion of a webpage?
I'm working with an ecommerce site that has many product descriptions for various brands that are important to have but are all straight duplicates. I'm looking for some type of tag tht can be implemented to prevent Google from seeing these as duplicates while still allowing the page to rank in the index. I thought I had found it with Googleoff, googleon tag but it appears that this is only used with the google appliance hardware.
Technical SEO | | bradwayland0 -
Google ranking my site abroad, how to stop?
Hi Mozzers, I have a UK based ecommerce site, that sells only to the UK. Over the last month Google has started ranking my site on foreign flavours of Google, so I keep getting traffic coming to my site from Europe, America and the far east that we could never sell to, and as a result bounce is going up and engagement is going down. They are definitely coming to the site from google searches that relate to my product type, but in regions I do not service. Is there a way to stop google doing this? I have the target set to UK in WMT, but is there anything else I can do? I worried about my UK ranking being damaged by an increasing overall bounce rate. Thanks
Technical SEO | | FDFPres0 -
Want to Target Mobile site for Google Mobile Version and Desktop Site for Google Desktop Version
I have ecommerce site with both mobile version and desktop version. Mobile version starts with m.example.com and full version starts with www.example.com I am using same content through out both site and using 301 redirection by detecting user agent vice-versa. My both sites are accessible to crawl by any google spider. I have submitted both sites's sitemap to GWT and mobile site having mobile sitemap xml, so google can easily recognize my mobile site. Is it going to help to rank my both sites as per my expectation? I need to rank for mobile site in Google mobile and ranking for desktop site in Google desktop version. Some of pages of my mobile site are started to appearing in Google desktop version. So how I can stop them to appear in Google desktop? Your comments are highly welcome.
Technical SEO | | Hexpress0 -
Google is keeping very old title tags in the SERPs for my site. How can I fix this?
Hi Around 6 months ago a site I work with changed its brand. One company became two. Despite changing the title when a new site went live around 6 months ago Google still picks up the old title for certain search results relevant to the old title. When a search result is relevant to the new title it shows that. It's very frustrating as we are trying to re-brand and do not want the old brand name showing for some very important search results. Thanks in advance for your help Paul
Technical SEO | | pauldoffman0 -
Webmaster Tools Site Map Question
I have TLD that has authority and a number of micro-sites built off of the primary domain. All sites relate to the same topic, as I am promoting a destination. The primary site and each micro-site have their own CMS installation, but the domains are mapped accordingly. www.regionalsite.com/ <- primary
Technical SEO | | VERBInteractive
www.regioanlsite.com/theme1/ <- theme 1
www.regioanlsite.com/theme2/ <- theme 2
www.regionalsite.com/theme3/ <- theme 3 Question: Should my XML site map for Webmaster Tools feed all sites off of the primary domain site map or are there penalties for this? Thanks.0 -
My site cannot be found by google at all
I don't know why but our company site can not be found by google at all. I have submitted to google webmaster, have social media point to, etc, Is there any reason for this? url for our website is www.bistosamerica.com Thank you
Technical SEO | | BistosAmerica0