Listing of all Google Indexed Pages
-
I started managing a site that has about 391,000 indexed pages. I want to get to the bottom of why there are so many in preparation for a ecommerce Migration and improving SEO. Anyone know of a tool? Many tools I have came across can only take 100 at a time. I would love to get them in excel or a database. I look forward to the suggestions.
-
Using site:yourdomain.com in Google, and then going to the end of the results and telling it to show you all of the results, is a good first start. It should get you enough to get an idea of why there are duplicated pages.
The Moz crawl can also help you figure it out, as often with ecommerce you'll have URLs for sorting products by price, name, pagination parameters, etc. We'll throw up a flag when we see a bunch of duplicate content or duplicate titles.
Also look for the easy stuff, such as non-www doesn't direct to www. Fix that, and you've cut your pages in half.
-
I may not be answering this correctly...
Are you looking for a list of URLs? If so, easy peasy to use screaming frog.
If it's all the pages Google has indexed, I don't really know and I'm sorry! However, I will come back to this thread to see if someone else has the answer for you, because I'm quite interested in it myself!!!
Best of luck,
Amelia
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Social pages not lining up with my website/listing
Hi! Newbie here... 🙂 On my Moz report and I have been told that my social sites are not lining up with my webpage. Any tips on getting this done? Also on my Moz Analytic report, I've added my social sites but only facebook is coming up... Any tips would be appreciated!!! Thanks!
Moz Pro | | fullerton0 -
Software for checking local listings?
Is there any software that you can enter in the name of a company, phone number, address, or website URL and get back a list of all the local search terms they are on the first page for?
Moz Pro | | ChaseCameron2 -
Why does SEOMoz only crawl 1 page of my site?
My site is: www.thetravelingdutchman.com. It has quite a few pages, but for some reason SEOMoz only crawls one. Please advise. Thanks, Jasper
Moz Pro | | Japking0 -
Not all pages are being crawled
I am set up on the PRO plan, I was under the impression that it would crawl up to 10,000 pages. My site has just over 200 pages, but whenever I am crawled it only crawls 121 pages. Is this normal? It's hard to know how reliable my data is because a significant amount of pages are missing.
Moz Pro | | KristinHarding0 -
Loss of Google AdWords API
Since August, we've had intermittent access to the Google Adwords API. Like others in this space, they've let us know that our access has been revoked. We've been working to restore this access, but unfortunately, we have not been able to do so. In the meantime we are working hard to find a suitable replacement that is both stable and a good fit for our customers’ needs. We wanted to let our PRO members know, first to apologize, and second to explain how this has impacted your PRO access. We understand how important it is to have all your keyword data in one place, and we're sorry for any disruptions you've experienced from not having this data source in our app.We do want to clarify that this change does not affect your data from Google Analytics. The products affected by this change are our **Keyword Analysis tool and the keyword details and **keyword analysis pages within our web app. We removed this keyword query volume data within PRO a little earlier this year. You can still generate reports, but the metrics previously pulled from the Google Adwords data source is no longer there.The good news is, we don't actually use the Google AdWords data in our calculation of keyword difficulty, so these reports are still 100% accurate—they're just missing the information we usually pull from the API, which you can find on the Google AdWords Keyword Tool here. We're so sorry about this, and understand how frustrating the loss of this data is. We're hoping to have an alternative data source soon, and we will keep you posted as we make progress. Please feel free to comment, ask questions, and get help by replying to this post. Thank you!
Moz Pro | | AaronWheeler7 -
Article Directory Page Rank 7
Hi Guys, Would I be correct in my assumption that if I submitted articles to this directory: www . thefreelibrary.com I would recieve a link back to my site of PR7. Could this be correct? Any help with this and any information on Article Directories worth submitting in would be great. Thank for looking, Craig
Moz Pro | | fenwaymedia1 -
Strange Ranking On 1st Page Of Google For Competitive Keyword
Hey all, So, I've noticed a site ranking on the first page in Google for a keyword that I have been targeting for years. The keyword is "mixtapes" and pretty competitive. The site, http://www.newyorkmixtapes.com is ranking around #4 in Google. It seemed odd to me so I looked into the site using some of the tools here. Open Explorer showed that they have minimal links. The actual site is pretty spammy also. Does anyone have any idea how this site ranked so high so fast? Is it ranking because I am located in New York? Does Google still put lots of weight on exact match domains? It's just been bothering me because it's a keyword that I put a lot of work into and then the site rolls around an ranks well for it with barely any backlinks. I just wanted to put it out there and see if I'm missing something obvious. Thanks everyone!
Moz Pro | | getrightmusic0 -
Campaign keywords (48) and on page report (17)
I go into my campaign keywords and have 48, but when I view my on page report only 17 show up? All keywords have been in there since last Friday except 2-26-11, 4 I added this Friday 3-4-11.
Moz Pro | | SmallFry340