My website's pages are not being indexed correctly
-
Hi,
One of our websites, which is actually a price comparison engine, facing indexing problem at Google.
When we check “site:mywebsite.com “, there are lots of pages indexed which are not from mywebsite.com but from merchants websites. The index result page also shows merchant’s page title. In some cases the title is from merchant’s site but when the given link is accessed it points to mywebsite.com/index. Also the cache displays the merchant’s product page as the last indexed version rather than showing ours.
The mywebsite.com has quite few Merchants that send us their product feed. Those products are listed on comparison page with prices. The merchant’s links on comparison page are all no-follow links but some of the (not all) merchant’s product pages are indexed against mywebsite.com as mentioned above instead of product comparison page of mywebsite.com
How can we fix the issue?
Thanks!
-
Yeah i was thinking the same....
The interesting thing is we've removed the redirect page a week ago and replaced it with javascript redirect code. is that a good practice?
-
Ah. Regarding #3: If you have a disallow in the robots.txt the search engines won't pick up the noindex. Ensure the noindex code is in place on the applicable pages, remove the disallow, and the pages should be removed after they're crawled. getting that relationship straightened out might help with some of the other things as well. Cheers!
-
Thanks Ryan for the response. We'll surely prevent crawling of search result pages. Please check below points too. Thanks!!!
- The cache page shows merchant product page in full version as well as in text-only version.
- The title shown on the result page is also of the merchant's product page title.
- One thing on the comparison price page is merchants are redirected to their respective websites, the links are nofollow, but redirect page is indexed even after having it on robots.txt and noindex on redirect page.
- The redirect page is indexed like mywebsite.com/redirect-50187889-0
- Comparison listing is not similar to internal search result page but result pages are crawl-able.
-
no iFrames being used.
-
Thumbs up to Don's rec. Also when you look at the text only cache what kind of page are you seeing, if any? Sometimes the site: search is a little inconsistent so you can try forcing the delivery of certain pages with the inurl: modifier. One last caveat that comes to mind is that if the comparison listing is similar to an internal search results page, Google may not ever list it, "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines." from: https://support.google.com/webmasters/answer/35769 Cheers!
-
How are you merchant prices / info being displayed on your site? From your site or using IFrames?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's the best way to test Angular JS heavy page for SEO?
Hi Moz community, Our tech team has recently decided to try switching our product pages to be JavaScript dependent, this includes links, product descriptions and things like breadcrumbs in JS. Given my concerns, they will create a proof of concept with a few product pages in a QA environment so I can test the SEO implications of these changes. They are planning to use Angular 5 client side rendering without any prerendering. I suggested universal but they said the lift was too great, so we're testing to see if this works. I've read a lot of the articles in this guide to all things SEO and JS and am fairly confident in understanding when a site uses JS and how to troubleshoot to make sure everything is getting crawled and indexed. https://sitebulb.com/resources/guides/javascript-seo-resources/ However, I am not sure I'll be able to test the QA pages since they aren't indexable and lives behind a login. I will be able to crawl the page using Screaming Frog but that's generally regarded as what a crawler should be able to crawl and not really what Googlebot will actually be able to crawl and index. Any thoughts on this, is this concern valid? Thanks!
Technical SEO | | znotes0 -
Gradual Drop in GWT Indexed Pages for large website
Hey all, I am working on SEO for a massive sports website. The information provided will be limited but I will give you as much context as possible. I just started digging into it and have found several on-page SEO issues of which I will fix when I get to the meat of it but this seems like something else could be going on. I have attached an image below. It doesn't seem like it's a GWT bug as reported at one point either as it's been gradually dropping over the past year. Also, there is about a 20% drop in traffic in Google Analytics over this time as well. This website has hundreds of thousands of pages of player profiles, sports team information and more all marked up with JSON-LD. Some of the on-page stuff that needs to be fixed are the h1 and h2, title tags and meta description. Also, some of the descriptions are pulled from wikipedia and linked to a "view more" area. Anchor text has "sign up" language as well. Not looking for a magic bullet but to be pointed in the right direction. Where should I start checking off to ensure I cover my bases besides the on page stuff above? There aren't any serious errors and I don't see any manual penalties. There are 4,300 404's but I have seen plenty of sites with that many 404's all of which still got traffic. It doesn't look like a sitemap was submitted to GWT and when I try submitting sitemap.xml, I get a 504 error (network unreachable). Thanks for reading. I am just getting started on this project but would like to spend as much time sharpening the axe before getting to work. lJWk8Rh
Technical SEO | | ArashG0 -
Does Index Status Goes Down If Website is Penalized by Google?
Hi Friends, Few of my friends told me that index status of a website goes down goes if it is penalized by Google. I can see my organic traffic has went down drastically after the Panda update on July 17th 2015 but index status still remains the same. So, I am bit confused. Any advice on this.
Technical SEO | | Prabhu.Sundar0 -
How to properly change your website's address in Webmaster Tools?
Hi There,We've launched a new website and as part of the update have changed our domain name - now we need to tell Google of the changes: Both sites were verified in Webmaster Tools From the old site's gear icon, we chose "Change of address" As part of the "Change of address" checklist Google presented, we added 301 redirects to redirect the old domain to the new one But now that the 301 redirects are in place, Google can no longer verify the old site And because it can no longer verify the old site, Google won't let us complete the change of address form How do we tell Google of the change of address in this instance - and has anyone else encountered this?CheersBen
Technical SEO | | cmscss0 -
My beta site (beta.website.com) has been inadvertently indexed. Its cached pages are taking traffic away from our real website (website.com). Should I just "NO INDEX" the entire beta site and if so, what's the best way to do this? Please advise.
My beta site (beta.website.com) has been inadvertently indexed. Its cached pages are taking traffic away from our real website (website.com). Should I just "NO INDEX" the entire beta site and if so, what's the best way to do this? Are there any other precautions I should be taking? Please advise.
Technical SEO | | BVREID0 -
Pages removed from Google index?
Hi All, I had around 2,300 pages in the google index until a week ago. The index removed a load and left me with 152 submitted, 152 indexed? I have just re-submitted my sitemap and will wait to see what happens. Any idea why it has done this? I have seen a drop in my rankings since. Thanks
Technical SEO | | TomLondon0 -
Website's stability and it's affect on SEO
What is the best way to combat previous website stability issues? We had page load time and site stability problems over the course of several months. As a result our keyword rankings plummeted. Now that the issues have been resolved, what's the best/quickest way to regain our rankings on specific keywords? Thanks, Eric
Technical SEO | | MediaCause0 -
Why this page doesn't get indexed?
Hi, I've just taken over development and SEO for a site and we're having difficulty getting some key pages indexed on our site. They are two clicks away from the homepage, but still not getting indexed. They are recently created pages, with unique content on. The architecture looks like this:Homepage >> Car page >> Engine specific pageWhenever we add a new car, we link to its 'Car page' and it gets indexed very quickly. However the 'Engine pages' for that car don't get indexed, even after a couple of weeks. An example of one of these index pages are - http://www.carbuzz.co.uk/car-reviews/Volkswagen/Beetle-New/2.0-TSISo, things we've checked - 1. Yes, it's not blocked by robots.txt2. Yes, it's in the sitemap (http://www.carbuzz.co.uk/sitemap.xml)3. Yes, it's viewable to search spiders (e.g. the link is present in the html source)This page doesn't have a huge amount of unique content. We're a review aggregator, but it still does have some. Any suggestions as to why it isn't indexed?Thanks, David
Technical SEO | | soulnafein0
