Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Capital - Antitrust Conspiracy
I think we all have heard about Thumbtack breaking the rules w/ badges. Getting deindexed, then getting a 100M injection from Google capital and having the penalties removed: https://techcrunch.com/2014/08/20/service-marketplace-thumbtack-raises-100m-round-led-by-google-capital/ Our primary competitor is a different marketplace backed by Google Capital. Does anyone know of any low frequency products (reliant on SEO) backed Google Capital that has not won out within search? (i.e. is there any hope of competing against a low frequency marketplace after they have Google Capital backing?)
Search Behavior | | MarketGrowth0 -
Google Analytics Tagging
Hi. I'm trying to figure out a solution to three questions one of my current clients has asked me in regards to Google Analytics tagging, and I'm unsure how to respond. Can anyone help? See below the questions, 1. In Google Acquisition > Overview, their paid media is reporting as "Other". They do not run any Google paid ads. They only run Facebook paid ads. Is there a way to update the source so that it says "Paid" versus "Other" within the default channel? The current solution was advised to create a channel group that the client has to then tick on overtime they want to see this data with the correct labeling. They would prefer to see it in the default. Is it just a matter of going into the *default channel, choosing the "Paid" option, and then specifying the source/medium that contains Facebook, CPC, or referral to be categorized under this channel? Or is it something else? *Aware that changes to the Default Channel are permanent changes and will change how new traffic is classified. 2. In Google Acquisition > Overview > Referral, the clients website is showing up as a referring domain, both the TLD and the subdomain. My understanding is that it should actually be reporting under the "Direct" channel. How do I correct this? Is it just a matter of updating the Direct channel to include those domains? Or do I need to update the settings? The domain's www. http: all 301 redirect to their https://domain.com and https://subdomain.domain.com. Within settings it has been specified as www.domain.com and URL is http:// - also noticed that Bot Filtering has not been checked, assuming this could mess up the analytic data if not define? Do you know? 3. Audience segmentation > The client wants to be able to define it's audience by shopping intent and informational intent. Is there a clear way to do this, for example, by keywords used, e.g. buy, product name, entry (shopping intent), versus e.g. non-purchase intent, entry to the blog, length of time on site (info intent). Would be happy to have a conversation about the last question, since I'm conscious that there are probably multiple ways to define this - thanks. To the group, thank you for readying my questions and helping me with these solutions - your time is appreciated and valued. Sincerely, Amanda
Search Behavior | | AmandaValle.Digital0 -
Search Analytics update in Google Webmasters Tools? Where can we find search queries bringing traffic to website?
I just got up and see Search Analytic's being updated today totally. Their is no option to see old reports. As Search Analytics only share 999 keywords.
Search Behavior | | csfarnsworth
Whats next now?
How can a webmaster finds all search queries bringing traffic to his website?
Any paid or free tool?
Google Analytic's > Acquisition > Search engine optimization > search queries will this area helps? Whole question revolves around. Any good tool that will help you find all the queries bringing traffic to my website?0 -
Google PageRank and google keyword ranks
Hi, I manage around 13 Clients for SEO and this last two weeks i have been having problems with google keyword ranking some of my clients went down almost 40 positions for some keywords other 15 other stayed the same and some just went down 1 or 5 positions. Somebody know what is happening with google? is it shuffling around positions of people? Any advice or how you do your SEO I'm willing to work together and share experiences and tactics on SEO to people that also share with me in order to build more strategies on this amazing market. Thanks.
Search Behavior | | computernc0 -
Why do all traffic curves show a "saw tooth" pattern in Google Analytics
Greetings Mozzers This sounds like a dumb question, but it's bothering me just the same and I would like to think what the fine community of SEO experts at MOZ thinks I have taken a look at the traffic curve for about 20 different sites in very different industries (home improvement, wedding party supplies, pet feeding systems) and it seems to me that they all show a saw tooth pattern, more or less like the one below.
Search Behavior | | Masoko-T0 -
Google reconsideration nightmare
Hello and thanks in advance The website has had a penalty on it for a while now, around 10 months, it was worked on by an agency who bought bad links to it but before then it was worked on by other agencies that may have done the same. I cleaned up as many bad link (according to many posts read) and filled for reconsideration and was told to get rid of a whole bunch of links which i did not know existed. Downloaded WMT links as instructed by Google admin person and contacted a heap of people which took a lot of man hours and cost us a fortune. Resubmitted and again was shown a handful of links by the Google admin person and told to contact and remove. The funny thing is that a few of them I disavowed in my list so they should not have pointed these out. I emailed back and showed that everything I could do was done and am happy to disavow any other link which they though violated their terms. This was not enough and I was told to show more efforts in removing links and then resubmit for reconsideration. I have done as much as I can on the website, I cannot see any more links which show violation, if there are some I am happy to remove but am now at a stage where i need direction from others to tackle this matter. Any advice would be helpful; I cannot start over from scratch as it's a brand and not a small website.
Search Behavior | | Benbug0 -
Can I use Google Analytics to find out actual times of visits during the day??
Hi, I'm a newbie at all this - I hope someone can help me. We're thinking of running time-specific offers to try and convert as many of our customer site visits as possible e.g. 15% discount if you call between, say, 2 and 5pm. It would be really helpful to me to find out what times of day people are visiting our site. I can't seem to find a way to do this on Google Analytics. Can anyone help? Thanks so much Sue
Search Behavior | | 3Amigos0 -
Google Location - Taking Away Our National Reach?
Hey, I was just noticing that we achieve #2 ranking on Google for one of our customers for one of their primary keyword phrases. But then I noticed the traffic analytics were not matching what we should expect from that keyword phrase. Then I noticed, in using "Chrome's Incognito Window", that our location was automatically selected for our main geographical city area. I then went and changed that location from Denver, to San Diego & Also Chicago, just to see what would happen, and I noticed we instantly dropped from #2 to #7 when changing our location. I don't know what my question is, but I guess I feel like that is preventing us from achieving the results we need to sell ecommerce products. Is there any info on this or suggestions anyone has on how to tackle this issue? It feels like Google is pulling the rug out from underneath our feet and trying to spread rankings more to localized areas, rather than offering someone the opportunity to capitalize on good rankings for a national audience. I understand why they would do it, and I don't say I disagree. But it just seems to affect our work as SEO's doesn't it? Since we can't be as effective for customers that have a global audience instead of strictly a localized one. I'm curious to see what people have to say about this issue. Thanks!
Search Behavior | | JerDoggMckoy0