Listing of all Google Indexed Pages
-
I started managing a site that has about 391,000 indexed pages. I want to get to the bottom of why there are so many in preparation for a ecommerce Migration and improving SEO. Anyone know of a tool? Many tools I have came across can only take 100 at a time. I would love to get them in excel or a database. I look forward to the suggestions.
-
Using site:yourdomain.com in Google, and then going to the end of the results and telling it to show you all of the results, is a good first start. It should get you enough to get an idea of why there are duplicated pages.
The Moz crawl can also help you figure it out, as often with ecommerce you'll have URLs for sorting products by price, name, pagination parameters, etc. We'll throw up a flag when we see a bunch of duplicate content or duplicate titles.
Also look for the easy stuff, such as non-www doesn't direct to www. Fix that, and you've cut your pages in half.
-
I may not be answering this correctly...
Are you looking for a list of URLs? If so, easy peasy to use screaming frog.
If it's all the pages Google has indexed, I don't really know and I'm sorry! However, I will come back to this thread to see if someone else has the answer for you, because I'm quite interested in it myself!!!
Best of luck,
Amelia
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Page
I just Check Crawl the status error with Duplicate Page Content. As Mentioned Below. Songs.pk | Download free mp3, Hindi Music, Indian Mp3 Songs http://www.getmp3songspk.com Songs.pk | Download free mp3, Hindi Music, Indian Mp3 Songs http://getmp3songspk.com and then i added these lines to my htaccess file RewriteBase /
Moz Pro | | Getmp3songspk
RewriteCond %{HTTP_HOST} !^www.getmp3songspk.com$ [NC]
RewriteRule ^(.*)$ http://www.getmp3songspk.com/$1 [L,R=301] But Still See that error again when i crawl a new test.0 -
Does Google really care about cheaters?
In analysing my competitors website I have discovered that they have over 600 fake pages that appear to have been created by some sort of internal search engine software, all pages are exactly alike except for the keyword and the words "no results found" on every page. I have discovered, elsewhere, that their duplicate content is abnormally high, and that their pages are stuffed with keywords.(in on page grader) I come out top in every section in Moz open site explorer, except internal links and total links. Yet they sit at number 1, while my site is number 2, I believe that copying their antics would surely make me fall foul of Google, and certainly would never risk it. but with the keyword only getting 30 searches a month, are google bothered about them doing this, should I resign myself to not being number 1 and is there anyone on this Forum who believes that I should use the same tactics, to get above them.
Moz Pro | | jefftracey1 -
Not all pages are being crawled
I am set up on the PRO plan, I was under the impression that it would crawl up to 10,000 pages. My site has just over 200 pages, but whenever I am crawled it only crawls 121 pages. Is this normal? It's hard to know how reliable my data is because a significant amount of pages are missing.
Moz Pro | | KristinHarding0 -
Duplicate page errors
I have 102 duplicate page title errors and 64 duplicate page content errors. They are almost all from the email a friend forms that are on each product of my online store. I looked and the pages are identical except for the product name. Is this a real problem and if so is there a work around or should I see if I can turn off the email a friend option? Thanks for any information you can give me. Cingin Gifts
Moz Pro | | cingingifts0 -
Can I do a campaign for just a page?
We've been doing a lot of building and work on just one category page, but when i try to put it in the campaign it won't let me do any url that has a sub folder like www.mainsite.com/keyword-page. I can only do www.mainsite.com, and when i select the other campaign options like root domain or sub folder, roger pops up with an error. Is anyone else having this problem?
Moz Pro | | anchorwave0 -
Page Authority vs Domain Authority
I'm using the site explorer to compare a potential clients site against 4 others, in an incredibly competitive market. Each of their competitiors has a higher page authority (on the home page) than their domain authority. This is untrue for the clients site. (which have much lower metrics all round) Any input as to what this means/says about their competitors who I would guess (looking at some of their backlink profiles) have done some failry widespread grey hat stuff in the past. (Though haven't we all 😉 )
Moz Pro | | FDC0 -
Too many links on a page - pull-down menus
SEOMoz is showing too many links on the page (www.ankinlaw.com). There clearly aren't 100 visible links on the page. Is it counting each page found on the (extensive) pull-down menu as a link? If not, where do all the links come from?
Moz Pro | | rarbel0