Listing of all Google Indexed Pages
-
I started managing a site that has about 391,000 indexed pages. I want to get to the bottom of why there are so many in preparation for a ecommerce Migration and improving SEO. Anyone know of a tool? Many tools I have came across can only take 100 at a time. I would love to get them in excel or a database. I look forward to the suggestions.
-
Using site:yourdomain.com in Google, and then going to the end of the results and telling it to show you all of the results, is a good first start. It should get you enough to get an idea of why there are duplicated pages.
The Moz crawl can also help you figure it out, as often with ecommerce you'll have URLs for sorting products by price, name, pagination parameters, etc. We'll throw up a flag when we see a bunch of duplicate content or duplicate titles.
Also look for the easy stuff, such as non-www doesn't direct to www. Fix that, and you've cut your pages in half.
-
I may not be answering this correctly...
Are you looking for a list of URLs? If so, easy peasy to use screaming frog.
If it's all the pages Google has indexed, I don't really know and I'm sorry! However, I will come back to this thread to see if someone else has the answer for you, because I'm quite interested in it myself!!!
Best of luck,
Amelia
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz page optimizer - avoid keyword stuffing
Hi Moz community, A quesiton about our dutch travel insurance website. Two weeks ago we had a ranking drop from 17 to position 35. The project is still fresh (we started in may this year). I quess penguin 4.0 didn't like our rapid linkbuilding. But we also receive a message in page optimization tool for keyword stuffing. When we analyze the homepage www.reisverzekering.net for reisverzekering (meaning travel insurance) Moz suggests to avoid keywords stuffing. But I dont understand, we don't use the keyword that much and Yoast doesnt give this message. How much key
Moz Pro | | remkoallertz0 -
PR2 vs Page Authority 65
Looking at a website that has a Google Toolbar rank of 2. Moz Authority is 65. Why would there be such a huge difference? Site has been around since 2000 but has not been updated. Mosst of the links are from 3000 links all owned by the same person.
Moz Pro | | Ebtec0 -
Mozscape index dosen't update
Last Mozscape index update: July 11, 2013. Next Mozscape index update: August26, 2013 Today is August26.Why mozscape doesnt updated?
Moz Pro | | vahidafshari451 -
Google updated their algorithm. How up to date is OpenSiteExplorer's Domain and Page authority?
Hi guys, as we all know, Google Panda update is here and it changed how they rank inbound links and the linkers domain and page authority. But when I look at OpenSiteExplorer's results, I cant see a difference in the page and domain authority. Why is that?
Moz Pro | | Uds0 -
Find pages containing broken links.
hi everyone, for each internal broken links I need to find all the pages that contain it. In the Seomoz report there is only a refferer link for each broken link, but google webmaster tools indicates that the dead link is present in many pages of the site. there is a way to have these data with SEOmoz or other software, in a csv report ? thanks
Moz Pro | | wwmind0 -
Why is my domain not being indexed by OSE?
My domain: http://www.seoproim.com/ is not being analyzed by OSE for whatever reason. GWT and YSE can see the inbound links.... Whats up?
Moz Pro | | LucasGarvin0 -
How do I find the most linked to page of a site?
I'm looking at a site for a potential link and am trying to find the most linked to page. The SEOmoz toolbar tells me the root domain (DA) is linked to by 660 root domains but the main URL (PA) is linked to by 38 root domains. I used open site explorer and got the same # of 38 root domains in the result. From the Top Pages tab, I clicked on the 2nd page down and the SEOmoz toolbar gives me 189 root domains linking to that page (PA). Then I ran a Linkscape report to see what that would say and I get 146 linking root domains. 1. Is this 2nd page down on OSE the most linked to page? 2. a. Is something off in these numbers?
Moz Pro | | Motava
b. How come OSE/Linkscape doesn't report the 660 root domains in the DA?0
