Listing of all Google Indexed Pages
-
I started managing a site that has about 391,000 indexed pages. I want to get to the bottom of why there are so many in preparation for a ecommerce Migration and improving SEO. Anyone know of a tool? Many tools I have came across can only take 100 at a time. I would love to get them in excel or a database. I look forward to the suggestions.
-
Using site:yourdomain.com in Google, and then going to the end of the results and telling it to show you all of the results, is a good first start. It should get you enough to get an idea of why there are duplicated pages.
The Moz crawl can also help you figure it out, as often with ecommerce you'll have URLs for sorting products by price, name, pagination parameters, etc. We'll throw up a flag when we see a bunch of duplicate content or duplicate titles.
Also look for the easy stuff, such as non-www doesn't direct to www. Fix that, and you've cut your pages in half.
-
I may not be answering this correctly...
Are you looking for a list of URLs? If so, easy peasy to use screaming frog.
If it's all the pages Google has indexed, I don't really know and I'm sorry! However, I will come back to this thread to see if someone else has the answer for you, because I'm quite interested in it myself!!!
Best of luck,
Amelia
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pages with Temporary Redirects on pages that don't exist!
Hi There Another obvious question to some I hope. I ran my first report using the Moz crawler and I have a bunch of pages with temporary redirects as a medium level issue showing up. Trouble is the pages don't exist so they are being redirected to my custom 404 page. So for example I have a URL in the report being called up from lord only knows where!: www.domain.com/pdf/home.aspx This doesn't exist, I have only 1 home.aspx page and it's in the root directory! but it is giving a temp redirect to my 404 page as I would expect but that then leads to a MOZ error as outlined. So basically you could randomize any url up and it would give this error so I am trying to work out how I deal with it before Google starts to notice or before a competitor starts to throw all kinds at my site generating these errors. Any steering on this would be much appreciated!
Moz Pro | | Raptor-crew0 -
My home page has an "A" rank in Moz and not ranking in Google.
My home page recently dropped from page one in Google to not being ranked for my top keyword. The page has an "A" ranking in MOZ for the keyword. Is there a way to find out the cause. I did have what looked like a duplicate page for a while 3 months ago when a domain was forwarding to my home page incorrectly. Appeared with second domain name instead of primary. Our business has been 95% through internet leads so quite an issue. Is there anyway to find out what is going on.
Moz Pro | | FredRoven0 -
Duplicate Page Title error for an eCommerce store !!
I currently launched my eCommerce startup hosted in Shopify and linked with MOZ. From my first Crawl Report I am getting 580 Duplicate Page Title i.e. all my Collection page have the same title. I have googled and have been checking the MOZ community but cannot find a fix to it. Some of the URL's are - http://www.onlypetstore.com/collections/all http://www.onlypetstore.com/collections/all?page=10 http://www.onlypetstore.com/collections/all?page=100 http://www.onlypetstore.com/collections/all?page=101 http://www.onlypetstore.com/collections/all?page=102 http://www.onlypetstore.com/collections/all?page=103 I am new to SEO and any suggestions will be a great help to me.
Moz Pro | | OnlyPetStore2 -
Campaign. Only 1 page is crawled
I have a campaign setup a couple weeks ago and noticed that only 1 page has been crawled. Is there something I need to do to get all pages crawled?
Moz Pro | | priceseo0 -
I know our business listed in Yahoo and medranks.com (for example). But my open site explorer report doesn't show those. however on their sites, I see the listing. Why is this?
I know our business listed in Yahoo and medranks.com (for example). But my open site explorer report doesn't show those links on the inbound report. however on their respective sites, I see the listing when I search for us. And the link does work..... Why is this? Why don't I see it on the open site report?
Moz Pro | | cschwartzel0 -
On-Page Optimization Report: How Are Keywords Chosen?
Apologies if this has already been covered 100 times! Last month I set up a new campaign, and so far the On-Page Optimization tool has only crawled and graded three of my pages so far. I assume it takes time for more pages to be covered? But, here's my real question: I see that the tool is giving my pages grades based on certain keywords, but the tool itself seems to be deciding which keyword to use in grading each page. To use a made-up example, my example has a page about leather gloves, a page about wool mittens, and a page about cotton mittens. The last one is supposed to be optimized for the keyword "cotton mittens," but the tool is grading it based on how well it's optimized for "wool mittens." I can go into the drop-down at the top of the page and change the keyword that the page is graded on, and that gives me a new grade, but only for that instance. The next week, the tool is back to giving the page an F for "wool mittens." Is that because the tool decides that "wool mittens" is the keyword for which the page has the best chance of ranking, no matter what my intentions are? is there any way to permanently tell the tool that I want the page to target "wool mittens" as its main keyword? Thanks in advance for your help!
Moz Pro | | ScottShrum0 -
Duplicate Page Titles and Content
The SeoMoz crawler has found many pages like this on my site with /?Letter=Letter, e.g. http://www.johnsearles.com/metal-art-tiles/?D=A. I believe it is finding multiple caches of a page and identifying them as duplicates. Is there any way to screen out these multiple cache results?
Moz Pro | | johnsearles0 -
Has MozTrust predicted fall in Google rankings???
The definition of MozTrust is:
Moz Pro | | driansmith
"MozTrust is SEOmoz's global link trust score. It is similar to MozRank but rather than measuring link popularity, it measures link trust. Receiving links from sources which have inherent trust, such as the homepages of major university websites or certain governmental web pages, is a strong trust endorsement." That being the case it is quite disturbing that a number of websites have been hit very badly by the latest Google algorithm changes that did indeed have very respectable MozTrust rankings. Can I ask whether anybody has carried out any sort of analysis of MozTrust versus negative impact in Google rankings? The prediction would of course be that websites suffering from a lower MozTrust would have been hit quite badly by these recent changes.0