Google is not indexing all URLs
-
My website have company and events profile from 200 countries. So it does have lots of URL. Earlier in August 2014, Google used to crawl 90% of URLs we submit. Thing goes wrong when we shifted from http to https. We lost traffic. But we are gaining it slowly. Main concern is that, It still does not indexed all submitted URLs. It have crawled merely 8% of all URLs submitted. site address is businessvibes.com
Any help would be appreciated.
-
Hi,
It sounds like something may be getting blocked or not crawled correctly, possibly because of the move from http --> https. I assume this is handled by the .htaccess file?
I would also check to see if there are https page issues on the site. If it's bit, you may have to do some digging around, but you can see this by using Google Chrome --> right click on the page --> inspect element --> console. This will show you the errors on the page.
-Andy
-
Hi Mossa,
Thanks for advice. We have already 2 millions of pages which are crawled. and good number of page views and session/day.
I will go with your point
-
You can see, almost all company and event profile have internal linking for other profile and events
-
We are doing that but we have pages in billion which is literally time taking as it have countries and events from 200 countries
-
It is very difficult to get backlining for billions of URLs, so branding and other activities are there to take care of that.
-
http to https have 301 redirection and all sitemap have been updated.
-
-
Before I start, why your website is using “_blank” even on internal links? This does not make any sense to me and I was literally pissed after three clicks.
Anyways, if you are interested in increasing the indexed pages here are the few things you could do:
- Improve internal linking of the website. Especially when possible link pages that are not indexed to the pages that are indexed.
- Get some social media shares (tweets) to pages that are not indexed.
- Build some quality links to the pages that are not indexed in Google.
Also, make sure that the 301 redirections from http tp https versions are correct and working and xml sitemap is updated.
If you do that I am sure that Google will not only index your pages but will also rank better from the desired key phrases.
Hope this helps!
-
Hi Martijn,
We have various sitemap
www.businessvibes.com/sitemap.companyprofile5.xml
www.businessvibes.com/sitemap.companyprofile6.xml
www.businessvibes.com/sitemap.companyprofile7.xml
And many More
-
Hi,
Do you have sitemaps that could provide some insights into your indexation rate for the data that you've submitted?
M.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Analytics & Google Sheets
Hi guys, I'm looking to use google analytics google sheets extension to pull data into a google sheet with metrics from other tools e.g. semrush, moz, etc. From my understanding with google analytics google sheet extension its designed to run reports and doesn't really allow you to add other metrics. Basically what i'm trying to do is this: https://www.useloom.com/share/c1e42bfa60bd46fca2b1120018969ce8 Any suggestions/advice on how to do this would be great. Cheers.
Reporting & Analytics | | jayoliverwright0 -
Google Tag Manager Classes
Does anyone know of classes or people who teach Google Tag Manager and how to use it effectively?
Reporting & Analytics | | JQC0 -
Any issues with Google impressions dropping in Webmaster Tools?
I'm seeing a drop in impressions across all my websites that are hosted at a certain location. Just wanted to make sure that it is not some reporting issue that others are seeing.
Reporting & Analytics | | tdawson090 -
Custom URL tracking for paid online ads
Hello! Our marketing team is testing paid online ad networks (banner ads, mobile ads and video ads) this year in addition to some traditional television advertising. We would like to track how effective these various ads are, and I'm wondering the best method to do so. In all cases, we're not setting up new pages for these ads to go to. They will simply go to an existing page on our site. For tracking purposes, here's what I'm wondering... If we set up custom URLs on our site (e.g. www.example.com/sports) then have the visitor immediately redirected to the intended page on our site (typically our home page), will Google Analytics show that custom URL in any report to see how much traffic flowed through it? Or will it not register because a page doesn't really exist for that URL? The biggest issue is with the banner ad network, which targets around ten categories of sites (sports, moms, entertainment, etc.). I know I'll be able to find the network listed as a referrer to see the total traffic the network sent us, but we would like to know specifically which category drove the visits. As such, would a custom URL for each category work, or will Google Analytics not pick it up because it's not hitting an actual page? For television ads, we are showing our URL as www.example.com/tv, which would then redirect to the intended landing page. In this case, it's a little more challenging because I won't have the referrer traffic to go by. But same problem... will GA recognize that URL and show it in Behavior > Site Content > All Pages? I hope this makes sense. If not please let me know and I'll try to explain. Thanks in advance,
Reporting & Analytics | | SmileMoreSEO
Erik0 -
Showing significant visit in google analytics
Hello Everyone, I was checking GA account of my site and I have found there are few keywords have significant number of visits in GA. These keywords not ranking even in hundred(google SERPs).I am not understanding from where these visitors coming .Please help me out. I am attaching screenshot of those keyword.Last but not least when i check source of these keywords source are google. Thanks aScZ8WT.png
Reporting & Analytics | | Alick3000 -
Google as referring domain
Hi all, a colleague asked a question, which I could not answer (never even noticed this "problem") 😞 When we are logged into our GA account and go the referring domains section, we find Google. I always thought that these visitors came via Google Image Search, but not all of them do. Most of them come via "/imgres", but some come via "/" (always thought that "/" was the homepage?), "/url" and "//" Maybe I am just stupid, but honestly I could not explain what these strings mean... or how these visitors landed on our site... Can you help me???
Reporting & Analytics | | accessKellyOCG0 -
Using Regex for Goals in Google Analytics
I’ve got a website with 20 different forms; and I’d like to track all 20 form completions with one Goal in Google Analytics. The completion-page URLs follow the pattern below (the website uses Drupal CMS) Form One: www.acme.com/node/2990/done?sid=651 Form Two: www.acme.com /node/2991/done?sid=785 Form Three: www.acme.com /node/2992/done?sid=1021 Form Four: www.acme.com /node/2993/done?sid=459 I believe that there is a way I can use regex so Google Analytics will track if any of these completion URLs appears. Looking for guidance how to format the regex statement.
Reporting & Analytics | | TopFloor0 -
Do Google penalise you for having too many 404's?
Hi There I have been doing some work reducing the number of 404's displayed in the Crawl Errors found in Googles Webmaster Tools. We had a lot of products that were no longer available so have now been removed to reduce the number of 404s that had been found. However, there are a number of URLs that have been crawled that do not exist on our website and have been flagged in the list of Crawl Errors. I want to know if Google will penalise us for this, perhaps affecting our quality score or if they can see that this is something out of our control. This site for example: http://sibd.com/com_offers_unique_gifts.html has generated a lot of truncated URLs on its site that link to pages that don't exist on our site: http://www.arenaflowers.com/flowers/pri… That is the exact link that it is trying to locate. Here is the report for that particular link. As you can see the content has been scraped by other sites which has spread the problem further. Pages that link to http://www.arenaflowers.com/flowers/pri.. URL Discovery Date
Reporting & Analytics | | ArenaFlowers.com
http://www.justsearchit.com.au/for_flowers_offers,3.html
Sep 12, 2011
http://sibd.com/offers_unique_gifts_for.html
Sep 11, 2011
http://sibd.com/offers_unique_gifts.html
Sep 11, 2011
http://sibd.com/com_offers_unique_gifts_for.html
Sep 11, 2011
http://www.flexfinder.com/flowers_offers_unique_gifts.html
Sep 10, 2011
http://sibd.com/offers_unique_gifts_with.html
Sep 10, 2011
http://sibd.com/com_offers_unique_gifts.html
Sep 10, 2011
http://sibd.com/of_flowers_offers_from.html
Sep 9, 2011
http://arama.frmpc.com/for_flowers_for_less_ltd_includes.html
Sep 9, 2011
http://arama.frmpc.com/flowers_for_less_than_do.html
Sep 9, 2011 I have spotted a lot of these and currently have around 3.3K 404s in total, a majority are from sites we don't control. Is there an acceptable number of 404s a site should aim for and is the above something we should address or are Google smart enough to work out that we can't fix this ourselves? Thanks! Sam.0