Google Indexing of Site Map
-
We recently launched a new site - on June 4th we submitted our site map to google and almost instantly had all 25,000 URL's crawled (yay!).
On June 18th, we made some updates to the title & description tags for the majority of pages on our site and added new content to our home page so we submitted a new sitemap.
So far the results have been underwhelming and google has indexed a very low number of the updated pages. As a result, only a handful of the new titles and descriptions are showing up on the SERP pages.
Any ideas as to why this might be? What are the tricks to having google re-index all of the URLs in a sitemap?
-
No problem, its actually really easy:
https://www.google.com/webmasters/tools/googlebot-fetch
Once you have selected your account, add the URL and then submit to index. I would do the homepage first and for that page, use the "Crawl this URL and its direct links" option. Then for the subpages do the "Crawl only this URL" option. It can also help to do the "Crawl this URL and its direct links" for any of your top level menu items to help speed things up.
"For example, i just checked a page and saw that some images weren't being indexed." Does your robots file allow specific access to those pages? If not, here is how you can set it to do so. This will also allow Google's partners to access your images. Add this to the bottom of your robots file:
User-agent: Googlebot-Image
Allow: /images/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /
User-agent: Mediapartners-Google*
Allow: /
Sitemap: http://www.YOURSITEHERE.com/sitemap.xml -
Thank you!! I'll take a look through the google resource. Also the site:domain search reviled 35,000 results.
The results are there, just not reindexed.
-
David,
Thanks for your response. This is exactly what we've seen with the initial spike in ranking and now with things settling down. I'll make sure the team has the crawl requests to daily (which I think it is).
For fetch as google - what's the best way that you've used this? For example, i just checked a page and saw that some images weren't being indexed. If I correct the issue, can I just use "Submit to Index"?
Thanks!!!!
-
In the 1000's of sites we have submitted, all show an initial spike in ranking and indexing before things settle down for the long haul. It seems like Google does a "best guess" scenario, before they take the time to fully crawl and analyze all of the URL's and rank them accordingly. As always, resubmit the pages through all webmaster tools (Bing too!) so that they are always aware of the most recent updates. If you are planning on updating the pages frequently, I would edit your crawl request to daily in your sitemap. They probably won't do it anyway, but you can try
Use the fetch as Google religiously when you update. It is your friend
-
Hi there
Did you read through Google's indexing resources?
I would also try doing a quick "site:yourdomain.com" and see how many pages Google pulls up - that's a more accurate representation of what's indexed from your site. This is reflected in the resource above:
"Sometimes the data we show in Index Status is not fully reflected in Google Search results." I suggest reading through the resource and also performing that search. Google indexing your sitemap is a waiting game, you're on the watch, just be patient!
Hope this helps! Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Inner pages of a directory site wont index
I have a business directory site thats been around a long time but has always been split into two parts, a subdomain and the main domain. The subdomain has been used for listings for years but just recently Ive opened up the main domain and started adding listings there. The problem is that none of the listing pages seem to be betting indexed in Google. The main domain is indexed as is the category page and all its pages below that eg /category/travel but the actual business listing pages below that will not index. I can however get them to index if I request Google to crawl them in search console. A few other things: I have nothing blocked in the robots.txt file The site has a DA over 50 and a decent amount of backlinks There is a sitemap setup also any ideas?
Technical SEO | | linklander0 -
Google Indexing Desktop & Mobile Versions
We have a relatively new site and I have noticed recently that Google seems to be indexing both the mobile and the desktop version of our site. There are some queries where the mobile version will show up and sometimes both mobile and desktop show up. This can't be good. I would imagine that what is supposed to happen is that the desktop version is the one that should be indexed (always) and browser detection will load the mobile version where appropriate once the user is on the site. Do you have any advice on what we should do to solve this problem as we are a bit stuck?
Technical SEO | | simonukss0 -
Google not index main keyword on homepage in 2 countries same language, rest of pages no problem
Hello, Two the same websites, two countries, same language http://www.lavistarelatiegeschenken.nl / http://www.lavistarelatiegeschenken.be The main keyword "relatiegeschenken" in top 10 of netherlands (steady position for 2 years) and in ** belgium** not in top 15****0 the main keyword "relatiegeschenken| but other keywords good positions, thats so strange I didn't understand and now every thing turned around suddenly: Now the main keyword "relatiegeschenken suddenly " not anymore in top 10 in the netherslandsits gone and other kewyords still good positions , now **main keyword suddenly in top 10 of belgium 2 years was not **other pages still ok. It are exactly the same websites and the same language. So double content But my programmer told me in google webmaster tools settings are right, so no problem with double content ? I really dont understand first main keyword in netherland in top 10 and in belgium not, now changed, now in belgium top 10 and not findable in the netherland on the main keyword. Maybe problem in code ? Maybe problems in code because websites are identical and active in two different countries wit same language ? No message about a penalty message in WMT, no spam links week i delete two strong but according to Linkdetox a bad links. I can not find a solution but its really important keyword that my customer want back in top 10 in netherland, like it was. All other positions and visitors are the same. Befor i have had this with belgium site, also main keyword google not index homepage. But suddenly no google show in belgium in top 10 Its turned around Kind regards, Marcel
Technical SEO | | Bossie720 -
Bing indexing at a tiny fraction of Google
I've read through other posts about this but I can't find a solution that works for us. My site is porch.com, 1M+ pages indexed on Google, ~10k on Bing. I've submitted the same sitemaps, and there's nothing different for each bot in our robots file. It looks like Bing is more concerned with our 500 errors than Google, but not sure if that might be causing the issue. Can anyone point me to the right things to be researching/investigating? Fixing errors, sitemap crawling issues, etc. I'm not sure what to spend my time looking into...
Technical SEO | | Porch0 -
Small business sites banned by google. Please help.
Hi. My 2 sites http://www.painterdublin.com and http://www.tilers-dublin.com were banned by google update in November 2012. Both were ranking fairly well in search results: painterdublin generating cca 600/month and tilers-dublin cca 300/month organic traffic from google. After update it is about 70% less. Is there anyone willing to take a look at my pages and give me some advice about what to do to improve the situation? thank you very much
Technical SEO | | jarik0 -
Any one worked on a sites.google.com/ website ?
Hi I have a client that has a sites.google.com/ website, Has anyone ever used one ? or had to do SEO on one ? Any help would be very much appreciated Thanks
Technical SEO | | tempowebdesign0 -
Walking into a site I didn't build, easy way to fix this # indexing problem?
I recently joined a team with a site without a) Great content b) Not much of any search traffic I looked and all their url's are built in this way: Normal looking link -> not actually a new page but # like: /#content-title And it has no h1 tag. Page doesn't refresh. My initial thought is to gut the site and build it in wordpress, but first have to ask, is there a way to make a site with /#/ content loading friendly to search engines?
Technical SEO | | andrewhyde0 -
Does Google index XML files?
Does Google or other search engines include XML files in their index? More specifically, I am wondering how Google knows the difference between an xml filetype and an RSS feed.
Technical SEO | | nicole.healthline0