Locating Duplicate Pages
-
Hi,
Our website consists of approximately 15,000 pages however according to our Google Webmaster Tools account Google has around 26,000 pages for us in their index.
I have run through half a dozen sitemap generators and they all only discover the 15,000 pages that we know about. I have also thoroughly gone through the site to attempt to find any sections where we might be inadvertently generating duplicate pages without success.
It has been over six months since we did any structural changes (at which point we did 301's to the new locations) and so I'd like to think that the majority of these old pages have been removed from the Google Index. Additionally, the number of pages in the index doesn't appear to be going down by any discernable factor week on week.
I'm certain it's nothing to worry about however for my own peace of mind I'd like to just confirm that the additional 11,000 pages are just old results that will eventually disappear from the index and that we're not generating any duplicate content.
Unfortunately there doesn't appear to be a way to download a list of the 26,000 pages that Google has indexed so that I can compare it against our sitemap. Obviously I know about site:domain.com however this only returned the first 1,000 results which all checkout fine.
I was wondering if anybody knew of any methods or tools that we could use to attempt to identify these 11,000 extra pages in the Google index so we can confirm that they're just old pages which haven’t fallen out of the index yet and that they’re not going to be causing us a problem?
Thanks guys!
-
It's cool. Sorry, the point I was making is that irrespective of what you search for the page that is returned is http://www.refreshcartridges.co.uk/advanced_search_result.php (with nothing after the .php) and as such the search results page couldn't spurn multiple pages which could be indexed by Google.
-
Hmm, I'm not too knowledgeable about php pages. Sorry!
-
Sorry, I'm not sure what happened to that bit.ly address - The actual address of the website is www.refreshcartridges.co.uk.
Ah, I see what you mean about the search results now however this hopefully shouldn't be an issue as for security (our web guy said something about injections) the URL that is returned irrespective of what is searched for is http://www.refreshcartridges.co.uk/advanced_search_result.php
Thanks again!
-
I can't get that link to work.
What I said before still applies with physical input (this is what I assumed when I said it).
For example, user inputs the words "snakes and dogs" and clicks search. The new URL is "www.yoursite.com/search?q=snakes and dogs" All these weird URL pages need noindex meta tags or Google will flag them as duplicate content because, for example, this page and the result for "dogs and snakes" generate almost the same page.
Does that make sense?
It is in Google's Webmaster Guidelines that you should noindex these pages. -
Many thanks for your input on this. I have actually looked at this through the HTML improvements section of GWMT however I am showing only a few dozen duplicated titles / descriptions and this is simply due to the product categories being almost identical (for example HP Deskjet 500 and HP Deskjet 500+)
-
Many thanks for your response. Our site is an eCommerce site that doesn't employ tags as such and our categories are all accounted for in the 15,000 page figure.
-
We did have this at the beginning of the year when we used a ?dispmode=grid and ?dispmode=list to change the way our results were displayed. This has been rectified however by us completely removing the option and any instances of dispmode present in the URL force a 301 to the correct master page. There are still a few hundred instances of this dispmode being present in the Google index but 99% of them have fallen out now.
I have checked and double checked and we don't seem to have any issues like this at present.
-
I'm not certain if this is the case as our search engine requires physical input in order to yield a result. I don't know if it helps but the URL is http://bit.ly/4Cogchww if you fancy taking a look
-
Thanks for your reply. Indeed our website does force www. if someone were to attempt to navigate to us without prefixing www.
-
Hi Chris,
Google Webmaster has a tool that helps identify duplicate HTMLs and maybe you can use that to see if the 11,000 pages are duplicate. IF they are, I am assuming they should have the duplicate Title Tag and etc. which the tool may discover.
-
Have you checked for instances where a page parameter is being seen as another version of the same page? One of the sites I work for had an issue a few months back where every instance of a product page was being flagged as duplicate content because of an oversight. We had one of our coders write a clause into the page where every time a page loaded with a parameter such as ?color=72 it would canonicalize it to the page minus the parameter. This decreased our duplicate content warnings quickly and effectively.
-
it could be that your tags and categories are considered individual pages and therefore creating their own permalink: ex: http:www.example.com/keyword, and http://www.example.com/tag/keyword and http://www.example.com/category/keyword. Another way would be to check the sitemaps you have in webmaster tools and compare those to each other. Just a suggestion.
-
Does your website force 'www.'?
Both yourdomain.com and www.yourdomain.com are separate sites and can have different pages spidered.
-
Be sure to try different combinations of 'site:www.domain.com' and 'site:domain.com'. They will all yield different results.
Sounds to me like you probably have an internal search engine that is generating search results pages based off the search term, and each different results page is a piece of duplicate content.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Understanding why our new page doesn't rank. Internal link structure to blame? + understand canonical pages more.
Hi guys. Sorry it's an essay...BUT, i think a lot of you will find this an interesting question. This question is in 2 (related) parts, and I imagine it would be an 'advanced' SEO question. Hoping you guys can help bring some real insight 🙂 Always amazed at the quality for this forum/ community. **Context... ** We had a duplicate content issue caused by this page and it's product permutations, so we placed canonical tags on all the product permutations to solve it. Worked a treat. However, we now have more **product ranges. **We now sell Diaries, Notebooks & Music books, which are clearly different from one another. So...we've placed canonical tags on all the product permutations leading back to the 'parent' theme. In other words, all the diary permutations 'lead back' to the diary page. All the notebooks permutations 'lead back' to the main notebook page. So on and so forth. Make sense so far? Context end..... Issue. Amazingly our Diary page outranks our notebook pagefor the search term 'Design your own Notebook'. The notebook page is well optimised for this search term, and the diary page avoids the word 'notebook' altogether (so no keyword cannibalisation going on). Possible reason? Our Diary page has a vast amount of internal links to it throughout our site. The notebook page has only a few. Could this be the issue? If so, what reading/ blogs/ content/ tools would you recommend to help understand and solve this problem? i.e) Better understanding internal link structure for SEO. 2nd part of the question (in the context of internal linking for SEO). When there are internal links to a page with a conical tag does that 'count' towards the 'parent page', or simply towards that specific page? I really hope that makes sense. If it's clear as mud just shout. Isaac. EDIT: All pages in question have been indexed since we added these changes to the site.
On-Page Optimization | | isaac6630 -
Will pushing a visitor to a conversion page hosted on a 3rd-party domain hurt the landing page ranking
Had an interesting question from a client. The client has a page that is optimized for a specific term. The goal of the page is to push users to sign-up for a trial. The trial registration (conversion) page is hosted by a third-party. Will pushing users to the conversion page cannibalize the SEO authority of the landing page. My reflexive answer is to say no, but now am not so sure.
On-Page Optimization | | infoblue0 -
Title Tag duplication.
Hi Guys/Gals, We do a lot of work in a very competitive space (personal injury) and are having an internal debate on the best way to implement title tags for new sites. We understand that keywords, title tags, etc., don't possess the power they once did, but we have yet to see conclusive proof of this in our space. The vast majority of competitors still rank very well for keyword focused content, title tags, etc., while having average link profiles and little content. We write a lot of content for our clients and want to know if someone can offer their opinion on the question that follows this example: "Top 5 Injuries Caused by T-Bone Collisions | Indiana Accident Lawyer" Would it seem repetitive or manipulative to construct title tags as shown, always placing "Indiana Accident Attorney" or "Indianapolis Accident Lawyer," or similar of at the end of each title tag? Thanks, gang!
On-Page Optimization | | Wayne760 -
Too many page links warning... but each link has canonical back to main page? Is my page OK?
The Moz crawl warns me many of my pages have too many links, like this page http://www.webjobz.com/jobs/industry/Accounting ...... has 269 links but many of the links are like this /jobs/jobtitles/Accounting?k=&w=3&hiddenLocationID=463170&depth=2 and are used to refine search criteria.... when you click on those links they all have a canonical link back to http://www.webjobz.com/jobs/industry/Accounting Is my page being punished for this? Do I have to put "no follow" tags on every link I do not want the bots to follow and if I do so is Roger (moz bot) not going to count this as a link?
On-Page Optimization | | Webjobz0 -
Dynamic pages on a static html pages websiite
Good evening everybody. I am new on SeoMoz that I find very helpful in my work. I am not a web developer so please excuse me if my technical language is poor.I have an issue that maybe you can help me to solve. I work for a company who has a website, which is old and very well positioned on google for the most important keywords for the field of the company I work for (dentistry). It is a website made some years ago and it's made of static html pages.I would like to add a section in my website where I can post articles, like a blog (I think of a wordpress-style website) with daily posts (so with a dynamic page). Is there a way to do this without modifying the structure of the website and without losing pages, urls and ranking? We're on the first google page for many keywords of interests in our city and it would be a great damage for us to lose those positions.Thank you very much!
On-Page Optimization | | adec0 -
Landing Pages
Howdy Guys, We currently have around 19 landing pages that are near enough identical for each make of car. The content on each page isn't identical but you can tell its a template. Do you think we should change this and just target models instead of makes. Thanks, Scott
On-Page Optimization | | ScottBaxterWW0 -
If I want to rank well on one keyword would it be better to optimize multiple pages on the website for the keyword or should I only optimize one page for that keyword?
If I want to rank well on one keyword would it be better to optimize multiple pages on the website for the keyword or should I only optimize one page for that keyword?
On-Page Optimization | | CustomOnlineMarketing0 -
Is there a SEO penalty for multi links on same page going to same destination page?
Hi, Just a quick note. I hope you are able to assist. To cut a long story short, on the page below http://www.bookbluemountains.com.au/ -> Features Specials & Packages (middle column) we have 3 links per special going to the same page.
On-Page Optimization | | daveupton
1. Header is linked
2. Click on image link - currently with a no follow
3. 'More info' under the description paragraph is linked too - currently with a no follow Two arguments are as follows:
1. The reason we do not follow all 3 links is to reduce too many links which may appear spammy to Google. 2. Counter argument:
The point above has some validity, However, using no follow is basically telling the search engines that the webmaster “does not trust or doesn’t take responsibility” for what is behind the link, something you don’t want to do within your own website. There is no penalty as such for having too many links, the search engines will generally not worry after a certain number.. nothing that would concern this business though. I would suggest changing the no follow links a.s.a.p. Could you please advise thoughts. Many thanks Dave Upton [long signature removed by staff]0