Exporting Google and Bing Search Results
-
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages.
Cheers
-
There is a chrome extension called "scrape similar" that is useful for doing small batches of stuff like this. However it does have a couple of limitations in that you have to view each page & google will not show you all pages of a large domain. However it is quite easy and effective for sites with under 1000 pages.
https://chrome.google.com/webstore/detail/mbigbapnjcgaffohmbkdlecaccepngjd
The process can be sped up using other tools. I use tool that is designed for black hat forum/comment spamming to do SERP scrapes like that. Even if I did such spamming (I don't), I don't actually think this is a very good tool to do it with. However it is rather good at scraping results from google. However, again you are limited to how many results Google/Bing choose to show you.
If you need a bigger list then log files might be the way to do. You can get a list of all crawled URLs for any particular agent (including the likes of googlebot) from your server logs. Some hosts limit the size of these, so it might be worth checking before you start. However the data does get collected. The downside here of course is that you need access to the logs.
Of course crawled is not the same as indexed. Once you have that list you might need a further step to see which is indexed. Possibly cross-referencing it against google analytics landing pages or querying the google cache for that page (SEOtools for Excel from Biels Bosma is good for this).
Similarly, if you have a definitive list of the URLs on site you could start with that list and query which are cached.
Harder than it seems isn't it? Hopefully one of those methods will put you on the right track.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Increase in Rankings, but search visibility is decreasing
I just started updating my site with the very basics, focus keyword, meta description, and page titles. I see that I am going up in my targeted keywords, however my search visibility has dropped from 3.34% to 2.78%. My assumption is that my keyword choice is a little off, however I see that I have increased in keyword rankings and dropped very little in what I was ranking for previously, plus the keyword volume seems relatively the same. In fact I've added far more keywords than I have dropped. Curious why my search visibility has dropped, however my keyword rankings only seem to increase.
Moz Pro | | kthomasd0 -
8 point decrease in DA, increase in search traffic- should I disavow links?
Would a decrease in DA reflect the need to disavow junk links? My branded keywords recently bumped down two spots, whereas the keywords which have been bringing me the most traffic have been unaffected. Site: Edwardsturm.com DA decrease of 8 over past 3 months. Keyword "Edward Sturm" recently moved to second spot.
Moz Pro | | Edward_Sturm0 -
Is pagerank still a ranking factor for Google?
Hello Friends, Just want to confirm that the announcement by Google officials for no more pagerank update (Oct 14) is only for the Google toolbar update but pagerank still exists and still a ranking factor for Google? http://searchengineland.com/google-toolbar-pagerank-finally-officially-dead-205277 Please help. Regards
Moz Pro | | chakraseo0 -
Best Way to Seemlessly Use Moz and Google Analytics
Hi All, I have Moz Pro and I am beginning to explore Moz Analytics. I currently use Google Analytics for pretty much everything and I am almost at expert level at using the GA interface so I'm ready to learn a new tool and have a second source of data! My question is: what's the best way to get the most of Moz Analytics and GA congruently? Off hand I know that Moz's keyword, rank, and link data, including competitive insights, is more comprehensive than GA. But how else can I use Moz and get the most out of the Moz Analytics? Looking to really dive in and discuss how we can make use of both! Thanks - Vee
Moz Pro | | Symmetri0 -
Does Google Reward Bad Links in Some Industries?
This week I've been diving into competitive research again for several clients in different industries, and unless the data in Open Site Explorer is wrong, it really does appear that Google continues to reward sites with unnatural links in certain industries, especially industries that are less tech-savvy (agriculture, finance, manual labor). I've combed through all the links carefully, to ensure there aren't some secret high-quality links lurking, and for some of these sites that are showing up in the top ten for search they do not even one real, high-quality link. Also, their onsite optimization is often way over the top and the user experience is nothing less than nauseating. Could it be a MozScape issue, or does Google continue to reward terrible sites and link structures in certain industries? I'd love to hear your thoughts.
Moz Pro | | newwhy0 -
Open Site Explorer results vs. Google Webmaster Tools results
I've been comparing the links to my domain that OSE and GWT show and GWT shows many more links than OSE. Can anyone explain the difference? Does Google report no follow links that OSE does not?
Moz Pro | | cartersnipes0 -
How to rank each page correctly in the search engines
Hello everyone, I am new to SEOmoz and SEO in general. I am trying to wrap my head around everything and one of the issues I am having trouble grasping is how each individual page on our website plays into ranking for certain keywords. I will explain in more detail below: 1.) We are not a blog, we have a pretty static website. 2.) We can add as many pages to our website as needed (for SEO purposes) 3.) Our site is fairly small, about 25 pages on the website. Now with that information above, here are our main questions & concerns: 1.) What is the best way to rank in the search engines for 50 of our best converting keywords with only 25 pages on our website? 2.) We have 2-3 competitors that rank in positions 1-3 for about 30 of our keywords we are trying to rank for. But no matter what you type in, for example, "cars" "buy cars" "buy a car online" "truck prices" etc, the results that the search engines display our their homepages. Not the other pages on their websites that actually rank for each one of those individual keywords. Is this normal? That is something that we would like for our rankings to do if we have any control over it. 3.) Should each page try to rank for a unique individual keyword? For example, when using the SEOmoz On-page Optimization tool, should I try ranking 2 pages for "buy cars online" or just 1 page? Say my homepage has an "A" grade for the keyword, "buy cars online" but so does my second page on my website. Are these pages essentially competing with each other? Or will the search engines pick their favorite page and display the one they like the best? Any help is greatly appreciated and I look forward to getting this cleared up. Thanks
Moz Pro | | threebiz0 -
Drastic Difference between Ranking in Seomoz Keyword Ranking and non-personalized Google US Search
Hi everybody, for some of our keywords we saw a huge drop in ranking in the Seomoz Dashboard, yet when performing a non-personalized Google US search we still hold our old positions. Now, I am worried whether this is an upcoming Google penalty or just some bug in the latest crawl. Did anybody ever experience something similar and how could we find out if we were indeed penalized by Google (as we did some major linkbuilding campaign for these keywords recently)? Cheers and a major thanks for your suggestions, Tobias
Moz Pro | | space560