Exporting Google and Bing Search Results
-
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages.
Cheers
-
There is a chrome extension called "scrape similar" that is useful for doing small batches of stuff like this. However it does have a couple of limitations in that you have to view each page & google will not show you all pages of a large domain. However it is quite easy and effective for sites with under 1000 pages.
https://chrome.google.com/webstore/detail/mbigbapnjcgaffohmbkdlecaccepngjd
The process can be sped up using other tools. I use tool that is designed for black hat forum/comment spamming to do SERP scrapes like that. Even if I did such spamming (I don't), I don't actually think this is a very good tool to do it with. However it is rather good at scraping results from google. However, again you are limited to how many results Google/Bing choose to show you.
If you need a bigger list then log files might be the way to do. You can get a list of all crawled URLs for any particular agent (including the likes of googlebot) from your server logs. Some hosts limit the size of these, so it might be worth checking before you start. However the data does get collected. The downside here of course is that you need access to the logs.
Of course crawled is not the same as indexed. Once you have that list you might need a further step to see which is indexed. Possibly cross-referencing it against google analytics landing pages or querying the google cache for that page (SEOtools for Excel from Biels Bosma is good for this).
Similarly, if you have a definitive list of the URLs on site you could start with that list and query which are cached.
Harder than it seems isn't it? Hopefully one of those methods will put you on the right track.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google+ Local Business Page not appearing
I have a local verified google + listing and recently just went to update it nicely. even though it has been around for months, however Google just can't seem to index it, does anybody know the reason why? I have tried " hard-searching" for it and it is not even indexed! However if i find hard enough, it will appear in google maps, and also my domain has been around for months, if not a year however under Moz Open Site explorer it indicates it is a super new domain and even alexa is not recognizing any data even though it has traffic. The only thing i have done is i have shifted servers two months ago. Does that translate to a new site? I hope not...
Moz Pro | | partyrocks0 -
404 errors, but not showing in Google analytics
In my SEOmoz errors there are over a dozen 404 errors listed. However, they are not showing up in Google analytics. How can I find the referring url for the ones showing up here?
Moz Pro | | sakeith0 -
SEO Yoast data export
Just thought I would give something back. (Is this the right place!) I use Wordpress with the excellent SEO Yoast plugin. I needed a way of extracting the focus keywords that I have entered onto my pages along with the url for use on the SEOmoz On-page Optimisation tool. So I created GetYoastData which outputs to the browser the required data (and a bit more) that can be saved into an csv (Excel) file. Hope you find it useful - Yes it's not polished and yes it might output a blank line now and again but it's fairly useful. http://deanandrews.uk/get-yoast-seo-data/
Moz Pro | | DeanAndrews0 -
Tried to find correlation between SEOMOZ statistic and google ranking.
I've made Google US search and than tried to compare SEOMOZ data for first few result. I've used Open Site Explorer and On-Page Report Card. On attached picture you can find this statistic. Pages are sorted in order as they appear in Google search. For me is big question why position 2 and 4 is so hight and 6 is only 6. And what position 6 should do to go up. Thank you. 2s9zlu8.jpg
Moz Pro | | ctam0 -
Configure parameter effect in google wmt to reduce overly dynamic urls
We are looking at a weatherforecast site with realtime information that is updated every 5 minutes. For this website many urls have 6 parameters The SEOmoz campagne found duplicate information and overly dynamic urls. Then we went to google wmt section url parameters and configured parameters like day, month, year (effect: none). The next weekly SEOmoz campagne showed a big reduction in duplicates and small reduction overly dynamic urls. How can we reduce these 'errors' further?
Moz Pro | | theonlinefactory0 -
Rankings Check and Googles Quality Guidelines
Hi, I am trialing SEOmoz at present. What i dont understand is how come the rankings check, which presumably involves sending requests to Google, doesnt contravine the Google quality guidelines and cause a website to be penalised. Sorry if being a bit dim - but if anybody could explain it would be really helpful cheers stayin
Moz Pro | | stayin0 -
Where do I find "URLs Receiving Entrances Via Search" and "Non-Paid Keywords Sending Search Visits" in Google Analytics?
These are two metrics highlighted in the "Organic Traffic Data" report in the PRO campaigns. Since this report is composed of data linked from Google Analytics, I bet there's a way to find this same information in GA. So...anyone know how to do that in Google Analytics? I want this information for some long-tail productivity/potential research.
Moz Pro | | jcolman0