Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Acquisition & Audience Segmentation
Hi. I'm trying to figure out a solution to two questions one of my current clients has asked me in regards to Google Analytics tagging, and I'm unsure how to respond. Can anyone help? See below the questions, 1. In Google Acquisition > Overview, their paid media is reporting as "Other". They do not run any Google paid ads. They only run Facebook paid ads. Is there a way to update the source so that it says "Paid" versus "Other" within the default channel? The current solution was advised to create a channel group that the client has to then tick on overtime they want to see this data with the correct labeling. They would prefer to see it in the default. Is it just a matter of going into the *default channel, choosing the "Paid" option, and then specifying the source/medium that contains Facebook, CPC, or referral to be categorized under this channel? Or is it something else? *Aware that changes to the Default Channel are permanent changes and will change how new traffic is classified. 2. Audience segmentation > The client wants to be able to define it's audience by shopping intent and informational intent. Is there a clear way to do this, for example, by keywords used, e.g. buy, product name, entry (shopping intent), versus e.g. non-purchase intent, entry to the blog, length of time on site (info intent). Would be happy to have a conversation about the last question, since I'm conscious that there are probably multiple ways to define this - thanks. To the group, thank you for readying my questions and helping me with these solutions - your time is appreciated and valued. Sincerely, Amanda
Search Behavior | | AmandaValle.Digital0 -
Google Analytics Tagging
Hi. I'm trying to figure out a solution to three questions one of my current clients has asked me in regards to Google Analytics tagging, and I'm unsure how to respond. Can anyone help? See below the questions, 1. In Google Acquisition > Overview, their paid media is reporting as "Other". They do not run any Google paid ads. They only run Facebook paid ads. Is there a way to update the source so that it says "Paid" versus "Other" within the default channel? The current solution was advised to create a channel group that the client has to then tick on overtime they want to see this data with the correct labeling. They would prefer to see it in the default. Is it just a matter of going into the *default channel, choosing the "Paid" option, and then specifying the source/medium that contains Facebook, CPC, or referral to be categorized under this channel? Or is it something else? *Aware that changes to the Default Channel are permanent changes and will change how new traffic is classified. 2. In Google Acquisition > Overview > Referral, the clients website is showing up as a referring domain, both the TLD and the subdomain. My understanding is that it should actually be reporting under the "Direct" channel. How do I correct this? Is it just a matter of updating the Direct channel to include those domains? Or do I need to update the settings? The domain's www. http: all 301 redirect to their https://domain.com and https://subdomain.domain.com. Within settings it has been specified as www.domain.com and URL is http:// - also noticed that Bot Filtering has not been checked, assuming this could mess up the analytic data if not define? Do you know? 3. Audience segmentation > The client wants to be able to define it's audience by shopping intent and informational intent. Is there a clear way to do this, for example, by keywords used, e.g. buy, product name, entry (shopping intent), versus e.g. non-purchase intent, entry to the blog, length of time on site (info intent). Would be happy to have a conversation about the last question, since I'm conscious that there are probably multiple ways to define this - thanks. To the group, thank you for readying my questions and helping me with these solutions - your time is appreciated and valued. Sincerely, Amanda
Search Behavior | | AmandaValle.Digital0 -
How does Google treat significant content changes to web pages and how should I flag them as such?
I have several pages (~30) that I have plans to overhaul. The URLs will be identical and the theme of the content will be the same (still talking about the same widgets, using the same language) but I will be adding a lot more useful information for users, specifically including things that I think will help with my fairly high bounce rate on these pages. I believe the changes will be significant enough for Google to notice, I was wondering if it goes "this is basically a new page now, I will treat it as such and rank accordingly" or does it go "well this content was rubbish last time I checked so it is probably still not great". My second question is, is there a way I can get Google to specifically crawl a page it already knows about with fresh eyes? I know in the Search Console I can ask Google to index new pages, and I've experimented with if I can ask it to crawl a page I know Google knows (it allows me to) but I couldn't see any evidence of it doing anything with that index. Some background The reason I'm doing this is because I noticed when these pages first ranked, they did very well (almost all first / second page for the terms I wanted). After about two weeks I've noticed them sliding down. It doesn't look like the competition is getting any better so my running theory is they ranked well to begin with because they are well linked internally and the content is good/relevant and one of the main things negatively impacting me (that google couldn't know at the time) is bounce rate.
Search Behavior | | tosbourn0 -
Anyone facing issues with Google Analytics today?
For some reason all pages on our site seem to be taking ages to load as they seem to be getting stuck at the "waiting for google-analytics.com"! Anyone else facing similar issues today?
Search Behavior | | prsntsnh0 -
Industry Benchmarks for Google Analytics
Hey Mozites! I was wondering if anyone has come across any good resources for industry benchmarks relating to google analytics? (Social stats would be great as well, but Google Analytics is my main concern) Specifically, I have a client that is a bank and I would love to be able to compare their stats (traffic, unique visitors, bounce rate, time on site, conversion rates...) to other regional banks. Any ideas? Thanks for reading and for your help!
Search Behavior | | iBecC0 -
Getting People to Leave Product Reviews, When they've reached our site in search of reviews
We have a product review website that has been showing a steady increase in organic traffic. However we seem to be having a disconnect between what happens when a visitor arrives. We essentially want a visitor coming to our website in search of a review, to LEAVE a review for another product they may have used... Any suggestions on ways we might encourage this? We are kind of newbies at this!
Search Behavior | | Sarah_Frantz0 -
Google analytics realtime reporting same keywords with different capitalization
I was just looking in my analytics and I saw something I have never seen before. Maybe its old news but its new to me. I have attached a screen shot. I have 2 keywords listed which are the same but are capitalized different scrabble dictionary Scrabble dictionary Does Google really consider these different? PlstfwR
Search Behavior | | cbielich0 -
Google Instant Search? REALLY?! Why is this the result???
First, this is one I thought I would never ask, but: Could this be true? I had noticed my MacBook Pro was eating some power and wanted to see if I could optimize my settings so to take advantage what battery life was available. (I swear that is the truth). So, I am at a Starbucks on someone and typed into Google search bar the following query - Best way to conserve - and, received the following instant search results before I could type - power on a macbook pro I am not putting it here as I want you to ask yourself what came up. I have attached a screenshot. My colleague arrived and without prompting she retried and got the same. So I am curious: Do you get the same result and why do you think that result is so predominant? This is funny. efuts.png
Search Behavior | | RobertFisher2
