Home Pages of Several Websites are disappearing / reappearing in Google Index
-
Hi,
I periodically use the Google site command to confirm that our client's websites are fully indexed.
Over the past few months I have noticed a very strange phenomenon which is happening for a small subset of our client's websites... basically the home page keeps disappearing and reappearing in the Google index every few days. This is isolated to a few of our client's websites and I have also noticed that it is happening for some of our client's competitor's websites (over which we have absolutely no control).
In the past I have been led to believe that the absence of the home page in the index could imply a penalty of some sort. This does not seem to be the case since these sites continue to rank the same in various Google searches regardless of whether or not the home page is listed in the index.
Below are some examples of sites of our clients where the home page is currently not indexed - although they may be indexed by the time you read this and try it yourself. Note that most of our clients are in Canada.
My questions are:
1. has anyone else experienced/noticed this?
2. any thoughts on whether this could imply some sort of penalty? or could it just be a bug in Google?
3. does Google offer a way to report stuff like this?
Note that we have been building websites for over 10 years so we have long been aware of issues like www vs. non-www, canonicalization, and meta content="noindex" (been there done that in 2005). I could be wrong but I do not believe that the site would keep disappearing and reappearing if something like this was the issue. Please feel free to scrutinize the home pages to see if I have overlooked something obvious - I AM getting old.
site:dietrichlaw.ca - this site has continually ranked in the top 3 for [kitchener personal injury lawyers] for many years.
site:burntucker.com - since we took over this site last year it has moved up to page 1 for [ottawa personal injury lawyers]
site:bolandhowe.com - #1 for [aurora personal injury lawyers]
site:imranlaw.ca - continually ranked in the top 3 for [mississauga immigration lawyers].
site:canadaenergy.ca - ranks #3 for [ontario hydro plans]
Thanks in advance!
Jim Donovan, President
-
I just took the first domain you gave me I tested them on two tools you lack canonical's on all but the homepage for all three and all three failed the https://varvy.com test
- imranlaw.ca
- dietrichlaw.ca
- canadaenergy.ca
- burntucker.com past the Varvy test but has only one canonical https://cl.ly/hPdN https://cl.ly/hPoe
- bolandhowe.com is the probably the most affected it has way too many 200 code URLs canonical's pointing to the HTTPS however they should be using a 301 redirect See search engine land post below & these photos https://cl.ly/hPyM & https://cl.ly/hPUj
Preform a search and replace see: https://cl.ly/hPe6
- https://searchenginewatch.com/sew/how-to/2291162/seo-audit-findings-4-hidden-technical-problems-that-can-send-dangerous-signals-to-search-engines
- https://searchenginewatch.com/sew/how-to/2300520/technical-seo-for-nontechnical-people
I took the domainIn number three above and ran it through screaming frog I found no canonical's for all but one URL. Take a look at what most of the URLs appear like.
In addition found that you have a redirect chain photos below they should go straight to HTTPS://www.canadaenergy.ca
I would utilize HSTS as well this will help considerably. And adding canonical's
https://cl.ly/hPJd to https://cl.ly/hPr1 to https://cl.ly/hPyj
Domain number two
the same situation you have one canonical URL homepage nothing else has a canonical
domain number one imranlaw.ca same situation see below no canonical except for the homepage
| Address | http://www.imranlaw.ca/ |
| URL Encoded Address | http://www.imranlaw.ca/ |
| Status Code | 200 |
| Status | OK |
| Content | text/html; charset=ISO-8859-1 |
| Size | 13160 |
| Title 1 | Mississauga Immigration Lawyer & Canadian Citizenship Attorney |
| Meta Description 1 | Imran Khan - Canada Immigration lawyer and Canadian Citizenship attorney Mississauga Immigration Lawyer |
| H1-1 | Canadian Immigration & Naturalization Lawyer |
| H2-1 | Imran Khan Law Office offers Legal Services in Immigration Law and Real Estate Law Matters. |
| Meta Robots 1 | index,follow |
| Canonical Link Element 1 | http://www.imranlaw.ca/ |
| Word Count | 275 |
| Level | 1 |
| Inlinks | 28 |
| Outlinks |19
|
| Address | http://www.imranlaw.ca/contact |
| URL Encoded Address | http://www.imranlaw.ca/contact |
| Status Code | 200 |
| Status | OK |
| Content | text/html; charset=ISO-8859-1 |
| Size | 14503 |
| Title 1 | Mississauga Immigration Lawyer - Contact |
| Meta Description 1 | Imran Khan - Canada Immigration lawyer and Canadian Citizenship attorney Contact Imran Khan |
| H1-1 | Contact Imran Khan Law Office |
| Meta Robots 1 | index,follow |
| Word Count | 276 |
| Level | 2 |
| Inlinks | 28 |
| Outlinks | 17 |A few domains the ones above which are listed below as well fail to be able to be seen by a synthetic Googlebot. Are you running them all on the same server?
You have some domains and in .com and others that end in .ca if you are looking in Google.ca and have geo-targeted the .com domains to Canada you should see them there. However if you're looking in Google.com obviously you cannot geo-target .CA domains to the United States therefore they would not show up in .com unless very rarely.
Deep crawl and screaming frog are going to be a best friends on this one. Please let me know if I can be of more help
here are my findings using a basic tool
and put it into https://varvy.com
The results were
Findable links
Well formed static links not found.
Page has no findable links.
Guideline states: 'Ensure that all pages on the site can be reached by a link from another findable page.'
Learn about links and site hierarchy
HTTP headers
Page headers when accessed as Googlebot.
Headers:
pages could not be found
https://varvy.com/hierarchyandlinks.html
Same thing for imranlaw.ca
Findable links
Well formed static links not found.
Page has no findable links.
Guideline states: 'Ensure that all pages on the site can be reached by a link from another findable page.'
Learn about links and site hierarchy
For canadaenergy.ca
Findable links
Well formed static links not found.
Page has no findable links.
Guideline states: 'Ensure that all pages on the site can be reached by a link from another findable page.'
Learn about links and site hierarchy
Amount of links
Amount of links not excessive.
0 links found on page.
Guideline states: 'Limit the number of links on a page to a reasonable number (a few thousand at most).'
Considering the amount of links on a page
**I wouldUse a tool like deepcrawl.com or screamingfrog.co.uk/seospider **
two determined exactly what is wrong with all three Domains which failed a very basic test of being able to be detected by Googlebot.
Hope this helps,
Tom
-
Hi Jim,
If analytics confirms that traffic is still landing on the homepage, then I think this is just Google reporting different pages when you perform a site: - It certainly doesn't sound like a penalty of any sort.
It is worth noting that Google did confirm some time back that site: doesn't bring back every page every time and is best used as a guide. Does the sitemap in Search Console show a healthy number of indexed links?
If you want a discussion on this, then it would be worthwhile also posting over at the Websearch Help Forums at Google and see what others have to say about it.
I hope this helps a little.
-Andy
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirect our www.website.com to website. Com/target-keyword
Dear moz community I have been analyzing the websites that rank in top 20 for our target keywords. All of the top 20 sites except us have their websites re directing to websites.com/target keyword. This is due to probably because they have multiple city's and one of the target keywords term is cityname + word. My question is and idea - make a 301 to our www.website.com to /city-keyword and start linking to that page with new links. Would that bring any benefit? Seems that it's a very strong ranking signal. Any threats that I must take into account? We currently rank as #9
Technical SEO | | advertisingcloud0 -
How to get google to forget my old but still working page and list my new fully optimized page for a keyword?
Hi There! (i am beginner in seo) I have dynamic and static pages on our site. I created a static page for a specific keyword. Fully optimized it, (h1, alt, metas, etc.....maybe too optimized). My problem is that this page is alive for weeks, checked it in GWT and it is in robots.txt, google sees it, and indexed it. BUT whenewer i do a search for that keyword, we still appear with the dynamically created link in the google listings. How could i "redirect" google, if sy make a search for that keyword than shows our optimized page? Is there a tool for that? I cant delete the dynamic page... Any ideas? Thx Andrew
Technical SEO | | Neckermann0 -
Crawl Test Report only shows home page and no inner site pages?
Hi, My site is [removed] When I first tried to set up a new campaign for the site, I received the error: Roger has detected a problem: We have detected that the root domain [removed] does not respond to web requests. Using this domain, we will be unable to crawl your site or present accurate SERP information. I then ran a Crawl Test per the FAQ. The SEOmoz crawl report only shows my home page URL and does not have any inner site pages. This is a Joomla site. What is the problem? Thanks! Dave
Technical SEO | | crave810 -
Google Page speed
I get the following advice from Google page speed: Suggestions for this page The following resources have identical contents, but are served from different URLs. Serve these resources from a consistent URL to save 1 request(s) and 77.1KiB. http://www.irishnews.com/ http://www.irishnews.com/index.aspx I'm not sure how to fix this the default page is http://www.irishnews.com/index.aspx, anybody know what need to be done please advise. thanks
Technical SEO | | Liammcmullen0 -
Does page speed affect what pages are in the index?
We have around 1.3m total pages, Google currently crawls on average 87k a day and our average page load is 1.7 seconds. Out of those 1.3m pages(1.2m being "spun up") google has only indexed around 368k and our SEO person is telling us that if we speed up the pages they will crawl the pages more and thus will index more of them. I personally don't believe this. At 87k pages a day Google has crawled our entire site in 2 weeks so they should have all of our pages in their DB by now and I think they are not index because they are poorly generated pages and it has nothing to do with the speed of the pages. Am I correct? Would speeding up the pages make Google crawl them faster and thus get more pages indexed?
Technical SEO | | upper2bits0 -
New Domain Page 7 Google but Page 1 Bing & Yahoo
Hi just wondered what other people's experience is with a new domain. Basically have a client with a domain registered end of May this year, so less than 3 months old! The site ranks for his keyword choice (not very competitive), which is in the domain name. For me I'm not at all surprised with Google's low ranking after such a short period but quite surprsied to see it ranking page 1 on Bing and Yahoo. No seo work has been done yet and there are no inbound links. Anyone else have experience of this? Should I be surprised or is that normal in the other two search engines? Thanks in advance Trevor
Technical SEO | | TrevorJones0 -
Redesign existing websites / worried about urls / mapping
Hi Guys, While redesigning existing websites that will have page name changes such as: example.com/products to be called example.com/solutions example.com/about-us to be called example.com/about should I 301 the old url to the new url. In the past I have not done this & I'm just wondering from an SEO point of view how bad is this? (On a scale of 1 to 10 how bad is this not 301ing urls, 10 being really bad & 1 being fine), Thanks.
Technical SEO | | Socialdude0 -
301 lots of old pages to home page
Will it hurt me if i redirect a few hundred old pages to my home page? I currently have a mess on my hands with many 404's showing up after moving my site to a new ecommerce server. We have been at the new server for 2 years but still have 337 404s showing up in google webmaster tools. I don't think it would affect users as very few people woudl find those old links but I don't want to mess with google. Also, how much are those 404s hurting my rank?
Technical SEO | | bhsiao1