Exporting Google and Bing Search Results
-
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages.
Cheers
-
There is a chrome extension called "scrape similar" that is useful for doing small batches of stuff like this. However it does have a couple of limitations in that you have to view each page & google will not show you all pages of a large domain. However it is quite easy and effective for sites with under 1000 pages.
https://chrome.google.com/webstore/detail/mbigbapnjcgaffohmbkdlecaccepngjd
The process can be sped up using other tools. I use tool that is designed for black hat forum/comment spamming to do SERP scrapes like that. Even if I did such spamming (I don't), I don't actually think this is a very good tool to do it with. However it is rather good at scraping results from google. However, again you are limited to how many results Google/Bing choose to show you.
If you need a bigger list then log files might be the way to do. You can get a list of all crawled URLs for any particular agent (including the likes of googlebot) from your server logs. Some hosts limit the size of these, so it might be worth checking before you start. However the data does get collected. The downside here of course is that you need access to the logs.
Of course crawled is not the same as indexed. Once you have that list you might need a further step to see which is indexed. Possibly cross-referencing it against google analytics landing pages or querying the google cache for that page (SEOtools for Excel from Biels Bosma is good for this).
Similarly, if you have a definitive list of the URLs on site you could start with that list and query which are cached.
Harder than it seems isn't it? Hopefully one of those methods will put you on the right track.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Anyone out there know MOZ well enough to provide a consultation on our results?
Hi Fans of MOZ, is there anyone out there that knows MOZ like an uber user and understands the details on crawl reports and what they mean? I'm looking for someone to review my MOZ results and give me some practical suggestions to work on with my SEO and web teams. Any takers?
Moz Pro | | NCPG
Bob Preston,0 -
Google+ Local Business Page not appearing
I have a local verified google + listing and recently just went to update it nicely. even though it has been around for months, however Google just can't seem to index it, does anybody know the reason why? I have tried " hard-searching" for it and it is not even indexed! However if i find hard enough, it will appear in google maps, and also my domain has been around for months, if not a year however under Moz Open Site explorer it indicates it is a super new domain and even alexa is not recognizing any data even though it has traffic. The only thing i have done is i have shifted servers two months ago. Does that translate to a new site? I hope not...
Moz Pro | | partyrocks0 -
Poor OSE results on my links
I am really disappointed with the "return on investment" I am getting from being a Pro member and using Open Site Explorer to analyze and remedy my inbound links post-penguin. My site has been crawled for over 4 months now by seomoz and OSE reprots that I have "external links" from just 25 or so domains - whereas Google webmaster tools has accurately identified well over 270 domains that link back to my site. How can do any sort of link removal strategy using open site explorer? I am paying $100/mo for SEomoz, and nothing for Google. What is the problem?
Moz Pro | | veezer0 -
How to increase ranking on google by using pro seomoz
Hi Team, I am trying to get good rank on this 2 keywords, gift card http://www.giftbig.com/gift-cards.html and gift vouchers http://www.giftbig.com/gift-vouchers.html, can anyonw suggest me how to rank on these 2 keywords. Now I m using SEO moz pro to get the solution.
Moz Pro | | Joydeep_das0 -
Exporting Twitter and FB data in report
Hi Been with this tool for a few days now and enjoying it so far. I do have one query though. In the campaigns section we have various tabs of data, including Social. However while all the other data is exportable in the created report, social is not available to add to the custom reports. Why is this? When I click on the social tab I can download it in CSV but it would be good to be able to export the charts in pdf as per the other analysis data. Would make it much easier when sharing reports with clients. Are there any plans to make the social metrics addable to the custom reports one can create?
Moz Pro | | GrumpyCarl0 -
Why have my rankings dropped on my keyword report but there is no change for the same keywords on google??
I received a keyword ranking report over the weekend, was horrified to see some of my keywords have dropped ranks by 20+ for some 😮 I checked them on google and they haven't moved (confused.com) Any information would be greatly appreciated.
Moz Pro | | Anthonykal-group0 -
Why are my rankings not showing up in Bing?
I recently signed up for SEOmoz and love it so far...but....in the keyword ranking section I know we rank in the top 20 for our keywords on Bing and yet for some reason the SEOmoz tool is coming back saying we arent even in the top 50. Why is this?
Moz Pro | | nextraq0 -
SEOMoz toolbar - Anyone else have problems with Search Profiles?
(Using Firefox 7.0.1) I just downloaded the toolbar and the Custom Search Profiles do not work--clicking on any of them adds "%" and numbers to the search query. I've created a couple of specific locations and I'd really like to get this figured out. Does this function work correctly for anyone? Am I doing something wrong?
Moz Pro | | Court_LOQUA0