Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Capital - Antitrust Conspiracy
I think we all have heard about Thumbtack breaking the rules w/ badges. Getting deindexed, then getting a 100M injection from Google capital and having the penalties removed: https://techcrunch.com/2014/08/20/service-marketplace-thumbtack-raises-100m-round-led-by-google-capital/ Our primary competitor is a different marketplace backed by Google Capital. Does anyone know of any low frequency products (reliant on SEO) backed Google Capital that has not won out within search? (i.e. is there any hope of competing against a low frequency marketplace after they have Google Capital backing?)
Search Behavior | | MarketGrowth0 -
Google Acquisition & Audience Segmentation
Hi. I'm trying to figure out a solution to two questions one of my current clients has asked me in regards to Google Analytics tagging, and I'm unsure how to respond. Can anyone help? See below the questions, 1. In Google Acquisition > Overview, their paid media is reporting as "Other". They do not run any Google paid ads. They only run Facebook paid ads. Is there a way to update the source so that it says "Paid" versus "Other" within the default channel? The current solution was advised to create a channel group that the client has to then tick on overtime they want to see this data with the correct labeling. They would prefer to see it in the default. Is it just a matter of going into the *default channel, choosing the "Paid" option, and then specifying the source/medium that contains Facebook, CPC, or referral to be categorized under this channel? Or is it something else? *Aware that changes to the Default Channel are permanent changes and will change how new traffic is classified. 2. Audience segmentation > The client wants to be able to define it's audience by shopping intent and informational intent. Is there a clear way to do this, for example, by keywords used, e.g. buy, product name, entry (shopping intent), versus e.g. non-purchase intent, entry to the blog, length of time on site (info intent). Would be happy to have a conversation about the last question, since I'm conscious that there are probably multiple ways to define this - thanks. To the group, thank you for readying my questions and helping me with these solutions - your time is appreciated and valued. Sincerely, Amanda
Search Behavior | | AmandaValle.Digital0 -
Google search operator "site:" show different result.
Search operator "site:" show incomplete information. When I search with just domain name it show only 3 link that got crawl in past week, this is the link https://www.google.com/?gfe_rd=cr&ei=mLr3VfrhN4_BuATQuYugBg&gws_rd=cr&fg=1#q=site:sierralivingconcepts.com&safe=off&tbs=qdr: but when i look a specific link it show them in any time (search tools), https://www.google.com/search?q=site:http://www.sierralivingconcepts.com/p-6300-white-silver-regence-louis-xiv-mango-wood-ornate-hall-console-table.aspx&safe=off&biw=1600&bih=775&noj=1&tbas=0&source=lnt&sa=X&ved=0CBUQpwVqFQoTCJ-4iOLK-McCFQFwjgod43gI9A But when i look in cached page it says "appeared on 11 Sep 2015" I am total confused why google not showing all the new link that it crawl from my site.
Search Behavior | | Sierra-Living-Concepts0 -
HUGE spike in Google Analytics Traffic
Hi there, I am witnessing a giant spike in my Google Analytics data (website: www.exchangecapital.com ) and I am completely stumped. My website usually gains roughly 15-20 visitors a day at most--and as of 11:10 am today my sessions for the day are up to 150. The traffic spike started on Friday at 132 sessions, Saturday at 261, Sunday at 247, etc. It's common that our sessions don't even hit the double digits over the weekends, so you can imagine my confusion. After trying to pin down some irregularities in geography, browser, and behavior, I'm still at a loss. I'm seeing a big spike in organic traffic (all not provided), as well as direct page visits, and I'm gaining traffic from US, Brazil, United Kingdom, Mexico, Spain, Malaysia, etc. etc--so not just one specific area. Is anyone else witnessing this in their data? Does anyone have any insight or ideas as to how I can look further into this? I am at a loss and any information would be appreciated. Thanks in advance! Lauren McLaughlin
Search Behavior | | LMcLaughlin0 -
Privacy Policy requirements for Demos and Interests in Google Analytics
I have a client who is activating the Demographics and Interests in Google Analytics, and I need to provide an appropriate privacy policy, per Google's TOS. Can anybody suggest what to state in the privacy policy? Google says: "If you’ve implemented Google Analytics Demographics and Interest Reporting, you must also disclose in your privacy policy: How you use data from Google's Interest-based advertising or 3rd-party audience data (such as age, gender, and interests) with Google Analytics." So, I guess it's advisable to state that the site gathers "3rd-party audience data." Is it enough to say that the info is being used to better understand and communicate with the website visitor?
Search Behavior | | jrae0 -
Google De-Indexed Our SIte for Branded Terms?
Hello all, As of 10am Pacific on September 12th, 2013, my team has noticed that our site, www.wirelessemporium.com, does not show up on the first 5 or 6 pages of SERPs for branded terms like "wireless emporium." We have not received any messages from Google via Webmaster Tools regarding this. Major activity that we've been doing to our site is updating content, meta tags, and h1 tags, along with removing/301 redirecting certain pages that did not meet Google quality guidelines. We've also been purging our backlink portfolio of toxic links and URLs, both manually and through the disavow tool. No blackhat has been done to this site for a very long time (more than 8 months now). One thing to note is that we did have a manual spam penalty placed on us back in July of 2012, it expired in early August of 2013 after a reconsideration request was submitted, and a 2nd manual spam penalty was placed on us again later that month. We are submitting a 2nd reconsideration request this Monday. Could this or the recent Panda update have anything to with this? We are very much in need of opinions as to why this is happening to our site. 5adbd14a31de3a78b998df94f0b6d2be
Search Behavior | | eugeneku0 -
Google reconsideration nightmare
Hello and thanks in advance The website has had a penalty on it for a while now, around 10 months, it was worked on by an agency who bought bad links to it but before then it was worked on by other agencies that may have done the same. I cleaned up as many bad link (according to many posts read) and filled for reconsideration and was told to get rid of a whole bunch of links which i did not know existed. Downloaded WMT links as instructed by Google admin person and contacted a heap of people which took a lot of man hours and cost us a fortune. Resubmitted and again was shown a handful of links by the Google admin person and told to contact and remove. The funny thing is that a few of them I disavowed in my list so they should not have pointed these out. I emailed back and showed that everything I could do was done and am happy to disavow any other link which they though violated their terms. This was not enough and I was told to show more efforts in removing links and then resubmit for reconsideration. I have done as much as I can on the website, I cannot see any more links which show violation, if there are some I am happy to remove but am now at a stage where i need direction from others to tackle this matter. Any advice would be helpful; I cannot start over from scratch as it's a brand and not a small website.
Search Behavior | | Benbug0 -
Recovering from a Hack: How long until Google reindexes changes?
In a previous post I made, I was able to determine that one of my sites; http://pokeronamac.com/ was hacked and was feeding spam perscription drug content to search engines, then redirecting to another site when clicked on Google. I then contacted my web host, and, after they did a scan of our files, they determined that something within the wp-includes directory was compromised and malicious. They removed the file, though they weren't able to determine the source of the attack, or how they god in (should we be scared?). Anyway, its been several days now ~5 and if I do a site search the spam pages still show up, but the redirect is no longer working. At this point, I am at a standstill, because i'm loosing traffic on my site by about 90%, and google hasn't sent us any warnings of malaware or the like. I know I was recommended against this before, but should I attempt to submit a reconsideration request, or should I just wait it out? Thanks for your help, Zach
Search Behavior | | Zachary_Russell0