Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Privacy Policy requirements for Demos and Interests in Google Analytics
I have a client who is activating the Demographics and Interests in Google Analytics, and I need to provide an appropriate privacy policy, per Google's TOS. Can anybody suggest what to state in the privacy policy? Google says: "If you’ve implemented Google Analytics Demographics and Interest Reporting, you must also disclose in your privacy policy: How you use data from Google's Interest-based advertising or 3rd-party audience data (such as age, gender, and interests) with Google Analytics." So, I guess it's advisable to state that the site gathers "3rd-party audience data." Is it enough to say that the info is being used to better understand and communicate with the website visitor?
Search Behavior | | jrae0 -
Vanish from google after choosing preferred domain and fixing 40 duplicate meta descriptions?
I recently followed 2 Google webmaster suggestions to clean up the on page SEO for our site. I chose a preferred domain 2 weeks ago(to www.website.com) and fixed the duplicate meta descriptions that our CMS was setting to unique and more natural descriptions for each page. I did that 3 days ago. Webmaster tools still says they are duplicates because it hasn't crawled the whole site yet. We have been fortunate enough to have some of our blog posts be covered by yahoo.com, cnet.com, huffingtonpost.com, gizmodo.com, etc. That is some major backlink juice and, as recently as 2 weeks ago, our website would be the #1 result when searching Google for "ourwebsite.com exact title of very popular blog post". Now it is on the 3rd page, and the top results are the websites that linked to our blog post. So....what gives? Is there a specific area I should look at? Our should I wait for Google to fully index our whole site now that changes have been made? It should be noted that our rankings have stayed the same in yahoo and bing. Any thoughts would be much appreciated.
Search Behavior | | garyislearning0 -
Are there better & inexpensive third party website analytics software over Google Analytics?
I've heard there are some third-party software that provide greater depth of information than Google Analytics, such as mouse tracking, heat mapping, video snapshots ect. Can anyone recommend a good program to use? I've tried a basic web-search but there seems to be a great variety of different ones.
Search Behavior | | Justin_hannan270 -
Google indexing PDF's
Hello, We work heavily on E-commerce SEO and recently Google has started to index PDF pages (Datasheets) added to the product pages instead of the actual product pages. Has anyone else noticed this at all? Seems to have got worse over the last month or so. Thanks
Search Behavior | | voipme0 -
How to best figure out synonymous search phrases
I'm curious about synonymous key phrases and competitive key term analysis. I have a few methods for conducting research to find what phrases Google considers synonymous but they are very time consuming. What are some ways to figure this out? An example would be livestock and cattle. Are there any good ways to figure out user intent related key phrases? For example in our niche if someone is looking to purchase something Google often delivers pages with title tags that include the term supplies. Would it be advantageous to optimize for these types of key phrases if the competition is less or should I assume that Google groups exact match and synonym and intent match together when calculating relevancy. Eg I can choose to target livestock or cattle on page. Livestock gets a quarter the search volume cattle gets but has half the competition. Will I be competing against the cattle sites too? If so should I just bite the bullet and target cattle?
Search Behavior | | Punkaj0 -
How to target local searches?
How do I start to target a national audiences? I currently get most of my hits from The US and Australia? But I see no reason why I don't get vies in Canada and The UK.
Search Behavior | | Stefan-Thorpe0 -
Google Query Contamination.... Are you seeing this??
I have been searching on Google.com with FireFox this morning. I searched for a state name.... like "georgia". Then I search for a product... like "guitars". For the Guitars query the first page of the SERPs include some music stores in georgia. So, Google is contaminating your search results with information from your most recent query. Are you seeing this too? Has Google been doing this for a long time? Or, I am going crazy?
Search Behavior | | EGOL3 -
Can we rank as a related search query on a competitor's brand keyword?
One of my clients wanted to know whether it's possible for him to rank as one of the related pages on Google for a brand name of a competitor. I honestly don't know whether it's possible so any ideas whether it's possible and if so, ideas on what can be done is greatly welcome. Thank you in advance. 🙂
Search Behavior | | als0070