Googlebot being redirected but not users?
-
Hi,
We seem to have a slightly odd issue.
We noticed that a number of our location category pages were slipping off 1 page, and onto page 2 in our niche. On inspection, we noticed that our Arizona page had started ranking in place of a number of other location pages - Cali, Idaho, NJ etc. Weirdly, the pages they had replaced were no longer indexed, and would remain so, despite being fetched, tweeted etc.
One test was to see when the dropped out pages had been last crawled, or at least cached. When conducting the 'cache:domain.com/category/location' on these pages, we were getting 301 redirected to, you guessed it, the Arizona page. Very odd. However, the dropped out pages were serving 200 OK when run through header checker tools, screaming frog etc.
On the face of it, it would seem Googlebot is getting redirected when it is hitting a number of our key location pages, but users are not. Has anyone experienced anything like this? The theming of the pages are quite different in terms of content, meta etc.
Thanks.
-
Can anyone offer any insight on this? The issue shows no sign of improving, and our organic is tanking.
If this really is an issue on Google's side, and it appears based on our technical set up, how can I force Google to take note and reindex the pages?
Here are some more examples, from across a number of categories..
Category A:
Page that has dropped out of index: https://goo.gl/KEQ8Yh
Cached version of that page: https://goo.gl/DPzFWMPage that has dropped out of index: https://goo.gl/5KiQ4s Cached version of that page: https://goo.gl/myRWNg Category B: Page that has dropped out of index: https://goo.gl/pr3YQs
Cached version of that page: https://goo.gl/8SEYi5 Page that has dropped out of index: https://goo.gl/LqzDrg
Cached version of that page: https://goo.gl/iwPs45 Category
Page that has dropped out of index: https://goo.gl/YBZS7c Cached version of that page:https://goo.gl/n33QzG Page that has dropped out of index: https://goo.gl/Ht4gfO
Cached version of that page: https://goo.gl/u81vbA -
Weirdly, at the time this seemed to correct itself overnight without any fixes our end. However 6 months on and it has happened again... Only this time, all location pages have dropped out of index, but latest cache versions all redirect to the Kansas page.
Have checked log files for these relevant URLs, and all show 200 (and being crawled regularly). When you fetch them in SC the show as complete, and they are all still indexed in Bing.
Very confused!
-
Hi Nicola,
Have you checked your canonical tags to be sure that they are pointing to the correct URL? Sometimes with storefront pages people just copy/paste the same HTML, swapping out the contact information. But maybe they forgot to update the canonical tags?
Keep us updated!
-
Thanks for advice Chris. Have checked, and can't see any issues with htaccess or robots seem. A very strange one indeed!
-
Hi Anthony,
Desktop returns 200, whilst mobile returns a 302 to the mDOT site.
Thanks.
-
Hi Nicola,
Sounds like a very strange one indeed. I can't say I've ever heard of Google giving a redirect when other 3rd party crawlers give you a 200.
My only suggestion to begin the troubleshooting would be to have a look at your htaccess and maybe Robots to see if there are any directives in there addressing Google specifically.
Anthony's suggestion is also a great idea. At least you'll see what Google is picking up on those pages right now rather what they saw last time it was cached.
-
You mention these pages are returning 200 in SEO tools, but would you please share what are these URLs are returning when you Fetch as Googlebot in Google Search Console? Test it in both desktop and mobile versions.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 Redirect in breadcrumb. How bad is it?
Hi all, How bad is it to have a link in the breadcrumb that 301 redirects? We had to create some hidden category pages in our ecommerce platform bigcommerce to create a display on our category pages in a certain format. Though whilst the category page was set to not visable in bigcommerce admin the URL still showed in the live site bread crumb. SO, we set a 301 redirect on it so it didnt produce a 404. However we have lost a lot of SEO ground the past few months. could this be why? is it bad to have a 301 redirect in the breadrcrumb.
Intermediate & Advanced SEO | | oceanstorm0 -
Redirection: Load balancer or CNAME?
We had a bunch of domains which no longer have sites tied to them but many have decent links pointing to them. In most cases we have other relevant content on live sites we can redirect these URL's to. We have been given the choice of redirection through the load balancer or direct as a cname on our CDN. I only have experience of 301's - What would be the preferred choice from an SEO perspective? Thanks, Sam
Intermediate & Advanced SEO | | Samsam00000 -
HTTPS 301 Redirect Question
Hi, I've just migrated our previous site (siteA) to our new url (siteB) and I've setup 301 redirects from the old url (siteA) to the new (siteB). However, the old url operated on https and users who try to go to the old url with https (https://siteA.com) receive a message that the server cannot be reached, while the users who go to http://siteA.com are redirected to siteB. Is there a way to 301 redirect https traffic? Also, from an SEO perspective if the site and all the references on Google search are https://siteA.com does a 301 redirect of http pass the domain authority, etc. or is https required? Thanks.
Intermediate & Advanced SEO | | opstart0 -
How to get rid of two 301 redirects?
I have two 301s from http://www. to https://non-www version of my site. I wonder how can get rid of one so it will look like this: 301-200 instead of 301-301-200 All other combinations work fine and give me 301-200 status codes. Thank you very much!
Intermediate & Advanced SEO | | lovemozforever0 -
Redirecting old mobile site
Hi All, Trying to figure out the best option here. I have a website that used to utilize a separate mobile site (m.xyz.com) but now utilizes responsive design. What is the best way to deal with that old mobile site? De-index? 301 redirect back to the main site in the rare case someone finds the m. site somewhere? THanks! Ricky
Intermediate & Advanced SEO | | RickyShockley0 -
How to find what Googlebot actually sees on a page?
1. When I disable java-script in Firefox and load our home page, it is missing entire middle section. 2. Also, the global nav dropdown menu does not display at all. (with java-script disabled) I believe this is not good. 3. But when type in <website name="">in Google search and click on the cached version of home page > and then click on text only version, It displays the Global nav links fine.</website> 4. When I switch the user agent to Googlebot(using Firefox plugin "User Agent Swticher)), the home page and global nav displays fine. Should I be worried about#1 and #2 then? How to find what Googlebot actually sees on a page? (I have tried "Fetch as Googlebot" from GWT. It displays source code.) Thanks for the help! Supriya.
Intermediate & Advanced SEO | | Amjath0 -
Googlebot crawling partial URLs
Hi guys, I've checked my email this morning and I've got a number of 404 errors over the weekend where Google has tried to crawl some of my existing pages but not found the full URL. Instead of hitting 'domain.com/folder/complete-pagename.php' it's hit 'domain.com/folder/comp'. This is definitely Googlebot/2.1; http://www.google.com/bot.html (66.249.72.53) but I can't find where it would have found only the partial URL. It certainly wasn't on the domain it's crawling and I can't find any links from external sites pointing to us with the incorrect URL. GoogleBot is doing the same thing across a single domain but in different sub-folders. Having checked Webmaster Tools there aren't any hard 404s and the soft ones aren't related and haven't occured since August. I'm really confused as to how this is happening.. Thanks!
Intermediate & Advanced SEO | | panini0 -
Htaccess Redirect with %C2%A0 in URL
Below is my setup for redirects in .htaccess file in my root word press installation. The www to non-www works well, so no problems there Other page redirects work well, too (example: redirect 301 /some-page/ http://mysite.com/another-page/ (I didn't post those because I have a few too many : ) So here it goes... RewriteEngine On
Intermediate & Advanced SEO | | pepsimoz
RewriteCond %{HTTP_HOST} ^www.mysite.com$ [NC]
RewriteRule ^(.*)$ http://mysite.com/$1 [R=301,L] BEGIN WordPress <ifmodule mod_rewrite.c="">RewriteEngine On
RewriteBase /
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]</ifmodule> END WordPress redirect 301 /archives/10-college- majors/ http://mysite.com/archives/10-college-majors/ redirect 301 /archives/10-college-%20majors/ http://mysite.com/archives/10-college-majors/ redirect 301 /archives/10-college-%C2%A0majors/ http://mysite.com/archives/10-college-majors/ I'm having a problem with the last 301 redirect: redirect 301 /archives/10-college-%C2%A0majors/ http://mysite.com/archives/10-college-majors/ not working... As you can see I've tried using other varations of the "space" but no go. I also used a redirect in cPanel's Redirect screen; testing all the possible options + wildcard I've also tried this: http://serverfault.com/questions/201829/using-special-characters-in-apache-mod-rewrite-rule (perhaps unsuccessfully, because it caused a 500 server error and it's a different situation in my case) I also saw something here: http://www.webmasterworld.com/apache/3908682.htm but I don't know if it works and how I would implement that + do so without compromising ALL other redirects. Note: the URL displays with a space in the address bar of all major web browsers: http://mysite.com/10-college- majors/ and goes to a 404 page I have a goregous page / PR6 / high authority site linking to the URL on my site, but they copied the URL with a space somehow. I contacted the person responsible for the website and he claims it works fine (aka he didn't check it). Is there a clean way to redirect ONLY this problematic URL without compromising other redirects, etc? Any ideas would be great. I'll respond with progress. Thanks in advance. UPDATE the redirect works, and it did work. Even so, when looking at source of page linking to mine, the URL looks like this: ``` http://mysite.com/archives/10-college- majors/ Clicking the URL in Source View in FireFox takes me to ``` http://mysite.com/archives/10-college-%C2%A0majors/ none of my 301 redirects should direct there. I don't have any redirect plugins either.0