Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ways to Identify Popular Search Terms
Certain searches seem intuitively like they would be popular, but don't appear so in my keyword research on Moz Pro. For example, I am a therapist and would have guessed that a lot of people would be searching for "online therapy California" during this pandemic, but actually those terms are not popular. I looked at Google Trends to see if I could understand this better, but It wasn't very helpful. Any other suggestions for where to get more information when search terms you would expect to be high volume don't appear to be so?
Search Behavior | | LPantell0 -
Google Index Issue - Indexing pages that don't exhist
Hi All, I have noticed a weird issue when performing a search on Google to show me all the pages it is indexing of our site. site:www.one2create.co.uk It brings up most of our website pages but then is also brings up a few HTTPS urls (our site has not been converted to HTTPS yet) but also the URL path, Title, and Meta Description are from one of our clients websites (an Automotive Job site). When clicked they take you to a generic 404 server error page, not our branded 404 page. The site that it has taken the url, title and meta description from is on a different server completely so I don't see how it has even managed to get that information and linked it to our site? Has anyone seen anything like this before? And what is the best way to fix it? We have asked Google to re-index the site but still no luck.
Search Behavior | | Jvickery0 -
Setting up Google Analytics
Hello, I will try to explain what I want and I am not sure if that is possible. Firstly I want to set up my Google Analytics that I would know where people come from and how many of them click on Google Ads. For example:
Search Behavior | | FCRMediaLietuva
Facebook > 2% of them click on ads.
Google > 3% of them click on ads.
Direct > 6% of them click on ads.Or whatever the results are. Secondly I would like to take people coming from facebook and divide them into age groups. For example: 100 ad clicks from facebook.
25-34: 9 clicks (9% of total facebook clicks, 2% of total ad clicks)
18-24: 7 clicks (.....
35-44: ... and so on Thirdly I would like to know which group use the adblocker the most.
Any other stuff that would be interesting if working with adsense? I believe that this would help me working with my target audience the best on facebook. Should this be set as goals? Sorry I am not an expert of Google Analytics 🙂 Thanks for the help.0 -
Google keyword planner shows low search volumes
Hello, There are some terms that you'd expect to see a lot of results. In fact, in the old days of Keyword Tool, I recall seeing thousands of results for a keyword like "anderson cooper". Today, I see a small amount as a monthly average. Am I doing something wrong? All I really want to do is find search volumes on particular keywords either globally or local to a country. And it's proving soo hard! 😞 Screen_Shot_2013_11_26_at_11_13_20_PM.png
Search Behavior | | mhamilton0 -
How to setup google analytics snapshot
Hello all! I get these awesome emails from google about a specific site. But I am not sure how, and why. But I like the data it generates. I am uploading a screen shot. They are called Analytics' Snapshot. My question is basically: How to setup google analytics snapshot to report on a specific website in my account? Because I have several I track - Over 40 domains, but I am mostly concered with like 4 -5 of them, I would like to report on. Thanks Mozers!! GJF3iQX
Search Behavior | | smstv0 -
From #2 TO Not Found in top 100 Results, WTF?
So my website was #2 for a targeted keyword on google. (6 months ago: # 😎 (3 months ago: #6) (1 week ago: #2). As you can see, it was moving up nicely, now it's not found on the top 100 search results. Only appears #2/(B) in the local search section. Could that be the reason? Websites that appear on the local search no longer appears in the SERP?
Search Behavior | | Droidman860 -
Keyword Search Historical data?
Is there a way to track the history of keyword search over time. I.E. the frequency of the Google search of the phrase 'unpasteurized milk' from 2006-2012 to see trending patterns.
Search Behavior | | preventionaid0 -
Googles 'freshness' update and how it may affect Estate Agents.
We have some clients who have raised concerns over how this new update will affect them. They are an Estate Agents who regularly update their property portfolio, blogs and news section etc. Can you advise how they may or may not be directly affected? Thank you.
Search Behavior | | SoundinTheory0