Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Capital - Antitrust Conspiracy
I think we all have heard about Thumbtack breaking the rules w/ badges. Getting deindexed, then getting a 100M injection from Google capital and having the penalties removed: https://techcrunch.com/2014/08/20/service-marketplace-thumbtack-raises-100m-round-led-by-google-capital/ Our primary competitor is a different marketplace backed by Google Capital. Does anyone know of any low frequency products (reliant on SEO) backed Google Capital that has not won out within search? (i.e. is there any hope of competing against a low frequency marketplace after they have Google Capital backing?)
Search Behavior | | MarketGrowth0 -
Ways to Identify Popular Search Terms
Certain searches seem intuitively like they would be popular, but don't appear so in my keyword research on Moz Pro. For example, I am a therapist and would have guessed that a lot of people would be searching for "online therapy California" during this pandemic, but actually those terms are not popular. I looked at Google Trends to see if I could understand this better, but It wasn't very helpful. Any other suggestions for where to get more information when search terms you would expect to be high volume don't appear to be so?
Search Behavior | | LPantell0 -
Which Search Engine Do Women Primarily Prefer?
I found a study in 2005 titled "Men Are From Google, Women Are From Yahoo" showing that women at that time preferred Yahoo as their primary search engine. I'm unable to track down updated statistical data to determine which search engine women primarily prefer. Any links to studies or resources would be greatly appreciated!
Search Behavior | | Prospector-Plastics1 -
Industry Benchmarks for Google Analytics
Hey Mozites! I was wondering if anyone has come across any good resources for industry benchmarks relating to google analytics? (Social stats would be great as well, but Google Analytics is my main concern) Specifically, I have a client that is a bank and I would love to be able to compare their stats (traffic, unique visitors, bounce rate, time on site, conversion rates...) to other regional banks. Any ideas? Thanks for reading and for your help!
Search Behavior | | iBecC0 -
New EU Laws governing cookies; will Google Analytics still be usable after May 25th?
Hi, first time I've posted a question, New EU Laws governing cookies; will Google Analytics still be usable after May 25th? - apologies in advance if its already been covered, but I couldn't find any answers when I searched - google search showed someone else ask the question already, but no straight answers were given:- From May 25th the Privacy and Electronic Communications Directive will come in to force which prohibits the use of cookies without informed, prior consent from web users. I've been trying to research to see whether or not this will affect Google Analytics.The Directive seems to be designed to prevent behavioural tracking, rather than web stats, but after reading the directive (with no law experience) I'm fairly confused by what will be prohibited. If anyone has any thoughts on the matter, I'd be very grateful! 🙂
Search Behavior | | bendyman1 -
Paid search impact on Organic Search traffic - Is it in the positive side or negative side?
Recently came across an article stating that Paid search may affect traffic through Organic search. Did anyone have similar experience?
Search Behavior | | gmk15670
