Google is forcing a 301 by truncating our URLs
-
Just recently we noticed that google has indexed truncated urls for many of our pages that get 301'd to the correct page.
For example, we have:
http://www.eventective.com/USA/Massachusetts/Bedford/107/Doubletree-Hotel-Boston-Bedford-Glen.htmlas the url linked everywhere and that's the only version of that page that we use.
Google somehow figured out that it would still go to the right place via 301 if they removed the html filename from the end, so they indexed just:
http://www.eventective.com/USA/Massachusetts/Bedford/107/
The 301 is not new. It used to 404, but (probably 5 years ago) we saw a few links come in with the html file missing on similar urls so we decided to 301 them instead thinking it would be helpful. We've preferred the longer version because it has the name in it and users that pay attention to the url can feel more confident they are going to the right place.
We've always used the full (longer) url and google used to index them all that way, but just recently we noticed about 1/2 of our urls have been converted to the shorter version in the SERPs. These shortened urls take the user to the right page via 301, so it isn't a case of the user landing in the wrong place, but over 100,000 301s may not be so good.
You can look at: site:www.eventective.com/usa/massachusetts/bedford/ and you'll noticed all of the urls to businesses at the top of the listings go to the truncated version, but toward the bottom they have the full url.
Can you explain to me why google would index a page that is 301'd to the right page and has been for years?
I have a lot of thoughts on why they would do this and even more ideas on how we could build our urls better, but I'd really like to hear from some people that aren't quite as close to it as I am.
One small detail that shouldn't affect this, but I'll mention it anyway, is that we have a mobile site with the same url pattern.
http://m.eventective.com/USA/Massachusetts/Bedford/107/Doubletree-Hotel-Boston-Bedford-Glen.html
We did not have the proper 301 in place on the m. site until the end of last week. I'm pretty sure it will be asked, so I'll also mention we have the rel=alternate/canonical set up between the www and m sites.
I'm also interested in any thoughts on how this may affect rankings since we seem to have been hit by something toward the end of last week. Don't hesitate to mention anything else you see that may have triggered whatever may have hit us.
Thank you,
Michael -
Lynn,
We had a few "site:" queries that we were watching as the full URLs came back replacing the truncated ones, for example: site:eventective.com/usa/Georgia/Atlanta. When we discovered the original problem, almost every listing page in those SERPs had a truncated URL, but by the start of last week it had gradually cleared up to only 6 or 7 listings with truncated URLs while all others had the full URL. Then suddenly we had 5 pages (50 listings) of truncated URLs and now almost 300 of them for that one query have the truncated version indexed. It appears to be continuing.
Another detail I noticed was in Webmaster Tools. All of our listings are in our sitemap with the full URL. When we had this problem before only about 50% of our pages listed in our sitemap were indexed, assuming that is because the truncated ones were in the index instead of the full URLs that were in the sitemap. As the truncated URL problem cleared up that ratio improved to the point where it was pretty steady at about 96-97% of our pages in our sitemap were indexed. Once this problem started to reappear that number dropped down to 90% and kept going down to the point where it is at 77% now.
The only real change we made was an upgrade to our server hardware at our hosting company.
I've considered disallowing the truncated URL pattern in the robots.txt, but I really shouldn't have to do that with the 301.
I'm starting to wonder whether google is sending us a signal that they like the shorter version of the URL better.
Thanks for taking the time to take a look at it.
Michael
-
Hi Micheal,
When you say you started noticing it again, this is through webmaster tools or through your own monitoring? I ask because having a look at the site I can see no technical reason why those truncated urls would be getting indexed again at first glance. Maybe it is just a matter of waiting a bit more for the last of them to get removed? If all of a sudden they have started creeping up again, it suggests some variable in the mix has changed again, but I cannot see anything that stands out.
-
Lynn,
Thanks again for helping us out with this back in May. After we made the corrections you pointed out it cleared up over the course of a few months. There were just a few truncated urls left until suddenly this week we noticed it starting again. I've looked at our 301s, our canonical/alternates, and made sure we are not linking to the truncated version anywhere, yet google continues to index the truncated version. I'm tempted to disallow the truncated version in my robots.txt file, but hesitate to do that because of the possibility of some unexpected side effects.
Do you or anyone else reading this have any idea why google would index:
http://www.eventective.com/USA/Massachusetts/Bedford/107/
rather than:
http://www.eventective.com/USA/Massachusetts/Bedford/107/Doubletree-Hotel-Boston-Bedford-Glen.html
when all links point to the latter and the former is even 301'd to the latter.
Any and all help is appreciated.
Thank you,
Michael
-
Lynn,
You nailed it. That's exactly what the problem was. Since we were using the same URL pattern for m. and www., we had created the canonical by swapping the "m" out of the current url and replacing it with "www". Since the truncated versions for mobile were in the index, they were all pointed to a truncated version for desktop.
As you pointed out, this should resolve itself over time. Now I can focus on just the ranking issue.
Thank you both Lynn and Jesse for your help.
Michael
-
Hi Micheal,
I suspect the mobile site might be responsible for the indexed urls issue. Your mobile site has loads of indexed pages with the shorter urls: https://www.google.com/#output=search&sclient=psy-ab&q=site:m.eventective.com&oq=site:m.eventective.com&fp=9861fb8dc6b3e7c
Before the 301 redirects on the mobile site were created, were the rel canonical links pointing to the truncated urls on the main site? Seems to be the case on this random page I grabbed:
So a kind of odd mixture of 301s on the main site, and a well indexed mobile site saying the rel canonical on the main site is the shorter url. Seems maybe the rel canonical won! Are you sure this is a recent issue? Maybe it has been like this for a while and just not noticed much?
I would think that with the 301s and rel canonicals now properly implemented on the mobile site then the index will slowly sort itself out. I suppose you could put a rel canonical on the main site page also referencing itself, might speed up the process a bit more.
Agree with Jesse that it is not likely a major worry and wouldn't think this alone would cause a ranking issue.
-
I'm responding to this in a semi-rushed matter as something is coming up but I just want to mention that the most likely reason for Google to index this version of your URL is because of the links pointing to it. Those which caused you to put a 301 in place, those that were 404ing prior... They are clearly demonstrating to be the authoritative URL to Google.
I'm not sure why you're worried about what the customer/user sees for URL. They are most likely looking more at the Title/Description in the SERPs well before the URL string. Most people only read the domain portion of a URL string and it's more used for the search engines purposes.. (my opinion) Also, once the user clicks your title or page they are taken to the redirect and the full URL string will be visible in the address bar of their browser.
As for why your rankings are affected... I'd be surprised if it had anything to do with this, honestly. If anything redirecting should help especially if you had links pointing to a broken page. The only exception would be if those links were poison, of course.
Okay got to run hope I was helpful. Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hit by an unnamed Google update on November 30th - Still suffering
Hi, Moz Community. Just decided to sign up for a free trial because I'm absolutely at my wits end here. Here's my site: cheapgamesguru.com I run a small PC gaming blog monetized by affiliate marketing. I do all the writing, SEO, etc. myself. The content I write is, from what I can tell, fully complying with Google's guidelines, and in and of itself is pretty informative and high-quality. My site was started in December of 2015, and it was doing very well for a good 10 or 11 months - until late November of 2016. Then something happened. My traffic started plummeting - I went from getting nearly 300 organic users a day (Not sessions - actual unique users) to 80, then 40, and now I'm lucky to get over 15 a day. I do not do ANY black hat SEO whatsoever. I have not taken part in any shady link building schemes, nor do I try to trick Google in any way. I just write good content, do good keyword research (Targeting only low-hanging fruit and low-difficulty keywords using KWFinder), and do my best to provide a good user experience. I run no ads on my site. Glenn Gabe wrote about a potential Google update on November 29th, but the stuff he said in his article doesn't seem to affect me - my mobile site is perfectly fine, according to Google's own metrics and testing tools. Here's the article in question: http://www.gsqi.com/marketing-blog/november-30-2016-google-algorithm-update/ At first, I thought it was possible that this was a result of my competitors simply doing far better than me - but that doesn't seem to be the case, as their rankings did not actually move - mine simply pummeted. And many of their sites are far worse than mine in terms of grammar, spelling, and site speed. I understand backlinks are important, by the way, but I really don't think that's why my site was hit. Many competitors of mine have little to no backlinks and are doing great, and it would also not make much sense for Google to hit an otherwise great site just because they have few backlinks. A friend of mine has reached out to Glenn Gabe himself to see if he can get his input on my site, but he's had a busy schedule and hasn't gotten a chance to take a look yet. I recently obtained a backlink from a highly relevant DA 65 site (About a month ago, same niche as my site), and it now shows up in Search Console and Ahrefs - but it hasn't affected rankings whatsoever. Important Note: I'm not only just ranking poorly for stuff, I'm ranking in position 100-150+ for many low-competition keywords. I have no idea why that is happening - is my site THAT bad, that my content deserves to be ranking on page 15 or lower? Sorry for the long question. I'm struggling here, and just wanted to give as much information as possible. I would really appreciate any input you guys can give me - if any SEO experts want to turn my site into a case study and work with me to improve things, I'd also be open to that 😉 I kid, of course - I know you guys are all busy. Thanks! P.S. I've attached a picture of my SEMRush graph, for reference, as well. mhgSw
Algorithm Updates | | polycountz0 -
How to follow up Google search in a Joomla website
Hello ! I'm facing a stupid issue but cannot solve it. Google Analytics can track the seach into a website. When I try to activate it I need to add some more parameters that I do not know. http://awesomescreenshot.com/03f1bnau8d Has anyone ever tried to configure Google analytics search for a Joomla website ? Tks a lot !
Algorithm Updates | | AymanH0 -
Google Update?
We have a website that for the past several weeks has been very consistent at between 13,500 and 14,200 daily visits and this site received 15,600 last Thursday. THIS week, Monday is at 22,200, Tuesday is at 26,200, and at mid-day today (at about our traffic halfway point in the day) we're already at 14,000 today. This was a site that was bringing about 14,000 visits as of May 16th last year and dropped to 11,000 the following week. The traffic to this site this week is so far beyond statistical analysis that there must have been something that happened.
Algorithm Updates | | sourcelinemedia0 -
Effect of new Google SSL policy on our Analytics - AACK!
So I went to look at our keyword reports in GA today and our most popular keyword was "(not provided)". It now accounts for 10% of our referred visits. Unfortunately, it also has a 125% avg order value compared to the rest of our site. This is a really annoying policy that Google has implemented and will clearly have an effect on our ability to effectively market our site.
Algorithm Updates | | IanTheScot0 -
How do I get the expanded results in a Google search?
I notice for certain site (ex: mint.com) that when I search, the top result has a very detailed view with options to click to different subsections of the site. However for my site, even though we're consistently the top result for our branded terms, the result is still only a single line item. How do I adjust this?
Algorithm Updates | | syount1 -
Removing secure subdomain from google index
we've noticed over the last few months that Google is not honoring our main website's robots.txt file. We have added rules to disallow secure pages such as: Disallow: /login.cgis Disallow: /logout.cgis Disallow: /password.cgis Disallow: /customer/* We have noticed that google is crawling these secure pages and then duplicating our complete ecommerce website across our secure subdomain in the google index (duplicate content) https://secure.domain.com/etc. Our webmaster recently implemented a specific robots.txt file for the secure subdomain disallow all however, these duplicated secure pages remain in the index. User-agent: *
Algorithm Updates | | marketing_zoovy.com
Disallow: / My question is should i request Google to remove these secure urls through Google Webmaster Tools? If so, is there any potential risk to my main ecommerce website? We have 8,700 pages currently indexed into google and would not want to risk any ill effects to our website. How would I submit this request in the URL Removal tools specifically? would inputting https://secure.domain.com/ cover all of the urls? We do not want any secure pages being indexed to the index and all secure pages are served on the secure.domain example. Please private message me for specific details if you'd like to see an example. Thank you,0 -
Google changing case of URLs in SERPs?
Noticed some strange behavior over the last week or so regarding our SERPs and I haven't been able to find anything on the web about what might be happening. Over the past two weeks, I've been seeing our URLs slowly change from upper case to lower case in the SERPs. Our URLs are usually /Blue-Fuzzy-Widgets.htm but Google has slowly been switching them to /blue-fuzzy-widgets.htm. There has been no change in our actual rankings nor has it happened to anyone else in the space. We're quite dumbfounded as to why Google would choose to serve the lower case URL. To be clear, we do not build links to these lower case URLs, only the upper. Any ideas what might be happening here?
Algorithm Updates | | Natitude0 -
Why would my product pages no longer be indexed in Google?
Our UK site has 72 pages in our sitemap. 30 of them are product pages which take a productid parameter. Prior to 1st Feb 2011, all pages were indexed in Google but since then all of our product pages seem to have dropped from the index? If I check in webmaster tools, I can see that we have submitted 72 pages and 42 are indexed. I realise we should have some better url structuring and I'm working on that but do you have any ideas on how we can get our product poages back into googles index http://www.ebacdirect.com
Algorithm Updates | | ebacltd0