Listing of all Google Indexed Pages
-
I started managing a site that has about 391,000 indexed pages. I want to get to the bottom of why there are so many in preparation for a ecommerce Migration and improving SEO. Anyone know of a tool? Many tools I have came across can only take 100 at a time. I would love to get them in excel or a database. I look forward to the suggestions.
-
Using site:yourdomain.com in Google, and then going to the end of the results and telling it to show you all of the results, is a good first start. It should get you enough to get an idea of why there are duplicated pages.
The Moz crawl can also help you figure it out, as often with ecommerce you'll have URLs for sorting products by price, name, pagination parameters, etc. We'll throw up a flag when we see a bunch of duplicate content or duplicate titles.
Also look for the easy stuff, such as non-www doesn't direct to www. Fix that, and you've cut your pages in half.
-
I may not be answering this correctly...
Are you looking for a list of URLs? If so, easy peasy to use screaming frog.
If it's all the pages Google has indexed, I don't really know and I'm sorry! However, I will come back to this thread to see if someone else has the answer for you, because I'm quite interested in it myself!!!
Best of luck,
Amelia
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does MOZ Provide an On Page Grader Embed?
I am looking to provide an on page grader on our website. Does MOZ provide this feature?
Moz Pro | | WebMarkets0 -
No page reports for one of my campaigns
Hi Everyone! so glad to be here, I had hardly visited the forum and been missing out. I have a client who hired me to set up their basic SEO for their website and I see no grades for the pages, at first I don't maybe the website was not being indexed but report shows it crawled 37 pages so what does it mean no grades what so ever, also showing now rankings, I put 30 to check and zilch. thank you.
Moz Pro | | FeuzaReis120 -
On Page URL's not updating?
I recently moved my site to Shopify which completely changed the URL structure. I went to my old pages and created permanent 301 redirects to my new pages on shopify. For some reason, the On Page reports are not picking up on one of the new pages. This is effecting my grading. Just wondering why this is happening & whether this may be an indication of a larger problem? Any help would be greatly appreciated! Thanks!
Moz Pro | | PedroAndJobu0 -
Unable to crawl pages
Hi, I am trying to set up a campaign for our website - www.salvationarmy.org.au however, I can't seem to get a scan of more than three pages. I have tried the following: www.salvationarmy.org.au (only 2 pages) www.salvationarmy.org.au/home (only 1 page) salvationarmy.org.au (only 3 pages) There is a geo IP redirect on www.salvationarmy.org.au but the second domain listed above should resolve the full site. I'm a newbie to SEOmoz so any help would be appreciated! Thanks, Mel
Moz Pro | | KingPings0 -
Truncate page URLs
We have some pages (for example a contact us form) for which the URL is modified by the CMS depending on the referring page (this helps to put the form submission in context for the sales reps who get the contact submission). The SEOmoz crawler considers each URL a new page -- and so numbers like in diagnostics are all inflated as the same page is listed multiple times (e.g. for too many links) Is there a setting to change what the crawler considers to be the same page? Here are two URLs for the same page that the reports treat as separate pages: http://www.spirent.com/About-Us/Contact_us.aspx?referurl=0F528F4D703D8BB3523738D6373AA8AD http://www.spirent.com/About-Us/Contact_us.aspx?referurl=10ACDA6055244E369395223437FDCF30 The page is actually: http://www.spirent.com/About-Us/Contact_us.aspx Thanks Ken
Moz Pro | | spirent.marcom0 -
Only 1 page has been crawled. Why?
I set a new profile up a fortnight ago. Last week seomoz crawled the entire site (10k pages), and this week has only crawled 1 page. Nothing's changed on the site that I'm aware of, so what's happened?
Moz Pro | | tompollard0 -
Multiple Anchor text Links to 1 Page
Hi, Currently we have this page on our site - http://www.highstreetvouchers.com/gift-vouchers/gift-vouchers-cards.jsp It contains both "Gift Vouchers " & "Gift Cards" content which are 2 keywords we are targeting for. Currently the top nav link anchor is "Gift Vouchers & Gift Cards" If we were to change the top nav to be "Gift Vouchers" & "Gift Cards" as different tabs, both linking to the same page, what might be the SEO consequences? thanks
Moz Pro | | NSJ780 -
Only Crawling 1 page?
Hi Guys, Any advice much appreciated on this! Recently set up a new campaign on my dashboard with just 5 keywords. The domain is brammer.co.uk and a quick Google site:brammer.co.uk shows a good amount of indexed pages. However - first seomoz tool crawl has only crawled 1 url!! "Last Crawl Completed: Apr. 12th, 2011 Next Crawl Starts: Apr. 17th, 2011" Any ideas what's stopping the tool crawl anymore of the site?? Cheers in advance.. J
Moz Pro | | lovealbatross0
