Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search Analytics update in Google Webmasters Tools? Where can we find search queries bringing traffic to website?
I just got up and see Search Analytic's being updated today totally. Their is no option to see old reports. As Search Analytics only share 999 keywords.
Search Behavior | | csfarnsworth
Whats next now?
How can a webmaster finds all search queries bringing traffic to his website?
Any paid or free tool?
Google Analytic's > Acquisition > Search engine optimization > search queries will this area helps? Whole question revolves around. Any good tool that will help you find all the queries bringing traffic to my website?0 -
What is the best way to learn google analytics?
Hi I'm trying to get my head around google analytics as a first time user. Can any suggest the best place to start after I have taken a look through the whole thing, how do I go about managing and tracking performances on my website? eg: Registrations/sign ups, conversions all that sort of thing Some advice will be much appreciated Thanks
Search Behavior | | edward-may0 -
Why do all traffic curves show a "saw tooth" pattern in Google Analytics
Greetings Mozzers This sounds like a dumb question, but it's bothering me just the same and I would like to think what the fine community of SEO experts at MOZ thinks I have taken a look at the traffic curve for about 20 different sites in very different industries (home improvement, wedding party supplies, pet feeding systems) and it seems to me that they all show a saw tooth pattern, more or less like the one below.
Search Behavior | | Masoko-T0 -
Huge difference in search results depending on the country?
I have some keywords ranking in pages 2 and 3 in the US but for instance, in Canada, the same keywords rank somewhere between pages 15 and 25. When I check my US rankings, I use a Windows VPS located in the US. When I check my rankings in Canada, I use a computer physically located in Canada. Basically, about 80% of my rankings have gone away for countries other than the US. I have not set any geographical target for my website in GWT. How come there is such a big difference in search results across different countries?
Search Behavior | | sbrault740 -
Why are brand, company name queries not showing up on Google SERP1?
I'm a bit confused today at Google's SERPs for a number of queries I've done. The first one that sparked my interest is when I searched for 'pinterest' (no quotes) and pinterest.com was NOT on the first page. In fact, the first pinterest.com result was on page 3 and was the page for somebody named Bil Chamberlain, who himself must be more related to the search term pinterest than pinterest.com itself. I've done this for a variety of search terms, signed out of google, signed in, and some companies' home page, or website in general comes up, and some do not. search for facebook, facebook.com is #1 search for united airlines, united.com's "Club and Lounges" page is on SERP 5. Am I missing something in google's search result methodology? I'd love to be directed to some article or discussion related to this because the last time I checked, most brand name or company name queries turned out company or brand home pages as reesult #1. I very well may just be behind the times. CCpjw
Search Behavior | | Buildings0 -
How do you know if you have been penalized by search engines?
How do you know if you have been penalized by search engines? I logged into my Google and Bing webmaster accounts and all looks ok, but in the last 30 days, my organic traffic has decreased 33%. My traffic usually steadily increases and if there is ever a decrease, it's usually very small.
Search Behavior | | StreetwiseReports0 -
Google Rel="Next" & Rel="Prev"
Hello, I have a catalogue website and I am implementing the rel="next" and rel="prev" to the website. My question is that we do have a view all page also, which apparently Google likes over a 'page1'.. Should I add the canonical to this page? I already have it set to WEBSITEURL/sonos (which is going well) I don’t want to have to change this to [URL]sonos/view-all (which is my view all link) as the first page is getting ranked well I am then telling Google no, the view all page is the parent. Any advice would be very much appreciated. Thanks Rick
Search Behavior | | Lantec0 -
Has anyone yet seen any Search or User benefits from implementing any Schema tags (as in Schema.org) ? Thanks
Schema tags have been around for a while, am really interested in knowing of any noticeable benefits seen by anyone who has had experience of implementing Schema tags.
Search Behavior | | SimonCullum0
