Google is forcing a 301 by truncating our URLs
-
Just recently we noticed that google has indexed truncated urls for many of our pages that get 301'd to the correct page.
For example, we have:
http://www.eventective.com/USA/Massachusetts/Bedford/107/Doubletree-Hotel-Boston-Bedford-Glen.htmlas the url linked everywhere and that's the only version of that page that we use.
Google somehow figured out that it would still go to the right place via 301 if they removed the html filename from the end, so they indexed just:
http://www.eventective.com/USA/Massachusetts/Bedford/107/
The 301 is not new. It used to 404, but (probably 5 years ago) we saw a few links come in with the html file missing on similar urls so we decided to 301 them instead thinking it would be helpful. We've preferred the longer version because it has the name in it and users that pay attention to the url can feel more confident they are going to the right place.
We've always used the full (longer) url and google used to index them all that way, but just recently we noticed about 1/2 of our urls have been converted to the shorter version in the SERPs. These shortened urls take the user to the right page via 301, so it isn't a case of the user landing in the wrong place, but over 100,000 301s may not be so good.
You can look at: site:www.eventective.com/usa/massachusetts/bedford/ and you'll noticed all of the urls to businesses at the top of the listings go to the truncated version, but toward the bottom they have the full url.
Can you explain to me why google would index a page that is 301'd to the right page and has been for years?
I have a lot of thoughts on why they would do this and even more ideas on how we could build our urls better, but I'd really like to hear from some people that aren't quite as close to it as I am.
One small detail that shouldn't affect this, but I'll mention it anyway, is that we have a mobile site with the same url pattern.
http://m.eventective.com/USA/Massachusetts/Bedford/107/Doubletree-Hotel-Boston-Bedford-Glen.html
We did not have the proper 301 in place on the m. site until the end of last week. I'm pretty sure it will be asked, so I'll also mention we have the rel=alternate/canonical set up between the www and m sites.
I'm also interested in any thoughts on how this may affect rankings since we seem to have been hit by something toward the end of last week. Don't hesitate to mention anything else you see that may have triggered whatever may have hit us.
Thank you,
Michael -
Lynn,
We had a few "site:" queries that we were watching as the full URLs came back replacing the truncated ones, for example: site:eventective.com/usa/Georgia/Atlanta. When we discovered the original problem, almost every listing page in those SERPs had a truncated URL, but by the start of last week it had gradually cleared up to only 6 or 7 listings with truncated URLs while all others had the full URL. Then suddenly we had 5 pages (50 listings) of truncated URLs and now almost 300 of them for that one query have the truncated version indexed. It appears to be continuing.
Another detail I noticed was in Webmaster Tools. All of our listings are in our sitemap with the full URL. When we had this problem before only about 50% of our pages listed in our sitemap were indexed, assuming that is because the truncated ones were in the index instead of the full URLs that were in the sitemap. As the truncated URL problem cleared up that ratio improved to the point where it was pretty steady at about 96-97% of our pages in our sitemap were indexed. Once this problem started to reappear that number dropped down to 90% and kept going down to the point where it is at 77% now.
The only real change we made was an upgrade to our server hardware at our hosting company.
I've considered disallowing the truncated URL pattern in the robots.txt, but I really shouldn't have to do that with the 301.
I'm starting to wonder whether google is sending us a signal that they like the shorter version of the URL better.
Thanks for taking the time to take a look at it.
Michael
-
Hi Micheal,
When you say you started noticing it again, this is through webmaster tools or through your own monitoring? I ask because having a look at the site I can see no technical reason why those truncated urls would be getting indexed again at first glance. Maybe it is just a matter of waiting a bit more for the last of them to get removed? If all of a sudden they have started creeping up again, it suggests some variable in the mix has changed again, but I cannot see anything that stands out.
-
Lynn,
Thanks again for helping us out with this back in May. After we made the corrections you pointed out it cleared up over the course of a few months. There were just a few truncated urls left until suddenly this week we noticed it starting again. I've looked at our 301s, our canonical/alternates, and made sure we are not linking to the truncated version anywhere, yet google continues to index the truncated version. I'm tempted to disallow the truncated version in my robots.txt file, but hesitate to do that because of the possibility of some unexpected side effects.
Do you or anyone else reading this have any idea why google would index:
http://www.eventective.com/USA/Massachusetts/Bedford/107/
rather than:
http://www.eventective.com/USA/Massachusetts/Bedford/107/Doubletree-Hotel-Boston-Bedford-Glen.html
when all links point to the latter and the former is even 301'd to the latter.
Any and all help is appreciated.
Thank you,
Michael
-
Lynn,
You nailed it. That's exactly what the problem was. Since we were using the same URL pattern for m. and www., we had created the canonical by swapping the "m" out of the current url and replacing it with "www". Since the truncated versions for mobile were in the index, they were all pointed to a truncated version for desktop.
As you pointed out, this should resolve itself over time. Now I can focus on just the ranking issue.
Thank you both Lynn and Jesse for your help.
Michael
-
Hi Micheal,
I suspect the mobile site might be responsible for the indexed urls issue. Your mobile site has loads of indexed pages with the shorter urls: https://www.google.com/#output=search&sclient=psy-ab&q=site:m.eventective.com&oq=site:m.eventective.com&fp=9861fb8dc6b3e7c
Before the 301 redirects on the mobile site were created, were the rel canonical links pointing to the truncated urls on the main site? Seems to be the case on this random page I grabbed:
So a kind of odd mixture of 301s on the main site, and a well indexed mobile site saying the rel canonical on the main site is the shorter url. Seems maybe the rel canonical won! Are you sure this is a recent issue? Maybe it has been like this for a while and just not noticed much?
I would think that with the 301s and rel canonicals now properly implemented on the mobile site then the index will slowly sort itself out. I suppose you could put a rel canonical on the main site page also referencing itself, might speed up the process a bit more.
Agree with Jesse that it is not likely a major worry and wouldn't think this alone would cause a ranking issue.
-
I'm responding to this in a semi-rushed matter as something is coming up but I just want to mention that the most likely reason for Google to index this version of your URL is because of the links pointing to it. Those which caused you to put a 301 in place, those that were 404ing prior... They are clearly demonstrating to be the authoritative URL to Google.
I'm not sure why you're worried about what the customer/user sees for URL. They are most likely looking more at the Title/Description in the SERPs well before the URL string. Most people only read the domain portion of a URL string and it's more used for the search engines purposes.. (my opinion) Also, once the user clicks your title or page they are taken to the redirect and the full URL string will be visible in the address bar of their browser.
As for why your rankings are affected... I'd be surprised if it had anything to do with this, honestly. If anything redirecting should help especially if you had links pointing to a broken page. The only exception would be if those links were poison, of course.
Okay got to run hope I was helpful. Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Internal pages ranking over the homepage: How to optimise to rank better at Google?
Hi, We have experienced a shift in SERP from internal pages ranking over website homepage for more than a year. Previously website homepages used to rank for the primary keyword like moz.com for "SEO". Now we can see that internal pages like moz.com/learn/seo/what-is-seo been ranking for the primary keyword "SEO". Google is picking up these "what is ABC" pages than the homepage. All our competitor sites are ranking with these internal pages which are about "what is (primary keyword)". We do have the same internal pages "what is....", but this pages is not ranking; only our homepage is ranking. Moreover we dropped more than 15 positions after this shift in SERP. How to diagnose this? Thanks
Algorithm Updates | | vtmoz0 -
Advice regarding latest Google Algo Update please, if possible...
Hi there I wonder if anyone can advise on this. Since the latest google update on 1st Sept or whenever it rolled out, we noticed an initial spike in hits on our site, which was great. However now we are noticing levels going back to where they were and less people visiting the site. It also seems to be very sporadic. So we have a period of say a couple of hours with no one on the site, then suddenly loads visiting. We have also noticed a big dip in enquiries, despite the site having roughly the same amount of visitors. All our stats on Moz, Webmaster Tools, Ahrefs, Serpfox and various other rank trackers are showing that we have had an increase in visibility on our tracked keywords. There is a definite spike on all, but where is our traffic and where are our enquiries? Usually we are able to work out where the problem is when updates occur but with this we have no idea. We are utterly baffled. Is this normal? Is this just fluctuations and will settle down? Has anyone else noticed weird things happening? If anyone has any ideas or experience of this then would be most grateful for any advice. Feeling rather desperate at the moment. Many thanks in advance. Clojo
Algorithm Updates | | Clojobobo0 -
What are the top tips for winning on Google, Bing, and Yahoo?
We just launched a new site that is starting to be indexed well in Google, but Bing and Yahoo are lagging a bit. I understand that the search engines are different algorithms and will take different lengths of time to index, rank, etc. What I'm curious about is are there any other tips / advice / things to keep in mind when trying to rank on the different search engines? Thanks!!
Algorithm Updates | | Emily_A0 -
Anchor name URLs & anchor blocks: how Google sees them?
Hi guys, Anchor name URLs & anchor blocks: how Google sees them? As far as I know Google hasn't ever recommended anchor name URLs and anchor blocks, mostly when you have one page site, but I have ran into an organic result with an hyper-link to an anchor name URL. anchor name link There is a proper link and there aren't on the page and the code the words "Jump to". It means Google has put those words there and it has also taken the header of that block as anchor text. Why has Google placed that link? The query is "faqs umbrella company", so I thought that Google has seen "faqs umbrella company" like "what is the most popular faq about umbrella companies?" and therefore perhaps the correct answer could be "Is an umbrella company the only option I have? What are the alternatives?". Although, IMHO the most popular FAQ on Umbrella Companies should always be "what is an umbrella company". Unfortunately, that page is only worthy of third Google organic result page and there is no hint of rich snippet or any kind of conversational/KBT optimisation on its source code. no-rich-snippet Someone has any idea of why Google shows that link and if it's something that we can optimise in our pages? Cheers Pierpaolo IhwGwkb.jpg VWORt5F.jpg
Algorithm Updates | | madcow780 -
Google penalty for one keyword?
Is it possible to get penalized by Google for a specific keyword and essentially disappear from the SERPs for that keyword but keep position for the brand (#1) and some other keywords (#4 and #7)? And how would you find out that this is what happened if there is no GWT message?
Algorithm Updates | | gfiedel0 -
Site name appended to page title in google search
Hi there, I have a strange problem concerning how the search results for my site appears in Google. The site is Texaspoker.dk and for some strange reason that name is appended at the end of the page title when I search for it in Google. The site name is not added to the page titles on the site. If I search in Google.dk (the relevant search engine for the country I am targeting) for "Unibet Fast Poker" I get the following page title displayed in the search results: Unibet Fast Poker starter i dag - få €10 og prøv ... - Texaspoker.dk If you visit the actual page you can see that there is no site name added to the page title: http://www.texaspoker.dk/unibet-fast-poker It looks like it is only being appended to the pages that contains rich snippets markup and not he forum threads where the rich snippets for some reason doesn't work. If I do a search for "Afstemning: Foretrukne TOPS Events" the title appears as it should without the site name being added: Afstemning: Foretrukne TOPS Events Anybody have any experience regarding this or an idea to why this is happening? Maybe the rich snippets are automatically pulling the publisher name from my Google+ account... edited: It doesn't seem to have anything to do with rich snippets, if I search for "Billeder og stuff v.2" the site name is also appended and if I search for "bedste poker bonus" the site name is not.
Algorithm Updates | | MPO0 -
Google Places......
Hi guys, Anyone come across a problem with Google places and it impacting organic search results? I have a client the was ranking top 3 on Google for all local keywords (letting agent). They have set up Google Places and now appear with a nice Google places ad but the organic position has now disappeared....Gone down to page 8 and nothing. Any ideas? Ta
Algorithm Updates | | SEOwins0 -
Is There Any Problem For Google When We Use Capital Letters in the Beginning of Each Word in TITLE?
I'm just wandering is there any difference when we use "Cheap Holidays to Egypt" or "Cheap holidays to Egypt". It is easier for users to read first option but would the second be more relevant for crawls?
Algorithm Updates | | fleetway0