Listing of all Google Indexed Pages
-
I started managing a site that has about 391,000 indexed pages. I want to get to the bottom of why there are so many in preparation for a ecommerce Migration and improving SEO. Anyone know of a tool? Many tools I have came across can only take 100 at a time. I would love to get them in excel or a database. I look forward to the suggestions.
-
Using site:yourdomain.com in Google, and then going to the end of the results and telling it to show you all of the results, is a good first start. It should get you enough to get an idea of why there are duplicated pages.
The Moz crawl can also help you figure it out, as often with ecommerce you'll have URLs for sorting products by price, name, pagination parameters, etc. We'll throw up a flag when we see a bunch of duplicate content or duplicate titles.
Also look for the easy stuff, such as non-www doesn't direct to www. Fix that, and you've cut your pages in half.
-
I may not be answering this correctly...
Are you looking for a list of URLs? If so, easy peasy to use screaming frog.
If it's all the pages Google has indexed, I don't really know and I'm sorry! However, I will come back to this thread to see if someone else has the answer for you, because I'm quite interested in it myself!!!
Best of luck,
Amelia
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does MOZ Provide an On Page Grader Embed?
I am looking to provide an on page grader on our website. Does MOZ provide this feature?
Moz Pro | | WebMarkets0 -
New pages on my web site
I have created web sites that appear somewhere on Google in hardly any time at all, but I appear to have forgotten something or things are different for pages added recently to an existing website. I have added a page on a particular subject, optimized it using on page grader, so that I get an A, and a check mark for everything except H1 tags and rel=canonical which my web hosting provider does not support. I do have a check mark for accessible to search engines The page has the format http://www.domain.com/specific-keyword It is in the menu, so should have internal links to it, as I understand it. I have created a new site map, and submitted it in webmaster tools. Interestingly it says that of the 96 pages only 76 were indexed is this a clue? and why would they not index a page I have then shared the page on google plus, facebook, tumblr, pinterest and twitter and some others In OSE it comes up as domain authority 28 page authority 1, the social media shares do show up in metrics on the right but no links internal or external are shown, they do on other pages I created in the same way. Is it just a case of waiting or is their something I do to help thank you
Moz Pro | | singingtelegramsuk0 -
Why is Moz Reporting as Duplicate Page Titles?
Our most recent MOZ crawl campaign is reporting 931 duplicate page title errors, most of which are "Product Review" pages like the following. Although there is only one review on this page, http://www.audiobooksonline.com/Cell_Stephen_King_unabridged_compact_discs.html, MOZ is reporting 15 duplicate page title, four of which I present below. http://www.audiobooksonline.com/reviews/review.php/full/0743554337/0/name/desc
Moz Pro | | lbohen
http://www.audiobooksonline.com/reviews/review.php/full/0743554337/0/rating/asc
http://www.audiobooksonline.com/reviews/review.php/full/0743554337/0/rating/desc
http://www.audiobooksonline.com/reviews/review.php/full/0743554337/0/state/asc Why is MOZ reporting these "pages" as duplicate page title errors? Are these errors hurting our SEO? How to fix?0 -
Seomoz legacy pages?
Hello, I am finding that I miss several of the old seomoz sections. The legacy tools in particular like the visual website comparison. Where is that now? Also, where is the ongoing list of the top 100 sites? So much was lost in the shift to MOZ, I hope some of the good old stuff is still available. Thank you, Nolan
Moz Pro | | QuietProgress0 -
Creating a campaign and connecting it to Google Analytics
I want to create three campaigns in Moz and connect them to my Google Analytics account. However, two of the domains I am trying to create campaigns for are joined in Google Analytics. Do I need to separate the two in Google in order to create separate campaigns for each in Moz?
Moz Pro | | ERKSPAL0 -
Campaign. Only 1 page is crawled
I have a campaign setup a couple weeks ago and noticed that only 1 page has been crawled. Is there something I need to do to get all pages crawled?
Moz Pro | | priceseo0 -
Does Google Reward Bad Links in Some Industries?
This week I've been diving into competitive research again for several clients in different industries, and unless the data in Open Site Explorer is wrong, it really does appear that Google continues to reward sites with unnatural links in certain industries, especially industries that are less tech-savvy (agriculture, finance, manual labor). I've combed through all the links carefully, to ensure there aren't some secret high-quality links lurking, and for some of these sites that are showing up in the top ten for search they do not even one real, high-quality link. Also, their onsite optimization is often way over the top and the user experience is nothing less than nauseating. Could it be a MozScape issue, or does Google continue to reward terrible sites and link structures in certain industries? I'd love to hear your thoughts.
Moz Pro | | newwhy0 -
Why did SEOMoz only crawl 1 page?
I have multiple campaigns and on a few of them SEOMoz has only crawled one page. I think this may have to do with how I set up the campaign. How do I get SEOMoz to crawl more than one page on these campaigns.
Moz Pro | | HermanAdvertising0