Exporting Google and Bing Search Results
-
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages.
Cheers
-
There is a chrome extension called "scrape similar" that is useful for doing small batches of stuff like this. However it does have a couple of limitations in that you have to view each page & google will not show you all pages of a large domain. However it is quite easy and effective for sites with under 1000 pages.
https://chrome.google.com/webstore/detail/mbigbapnjcgaffohmbkdlecaccepngjd
The process can be sped up using other tools. I use tool that is designed for black hat forum/comment spamming to do SERP scrapes like that. Even if I did such spamming (I don't), I don't actually think this is a very good tool to do it with. However it is rather good at scraping results from google. However, again you are limited to how many results Google/Bing choose to show you.
If you need a bigger list then log files might be the way to do. You can get a list of all crawled URLs for any particular agent (including the likes of googlebot) from your server logs. Some hosts limit the size of these, so it might be worth checking before you start. However the data does get collected. The downside here of course is that you need access to the logs.
Of course crawled is not the same as indexed. Once you have that list you might need a further step to see which is indexed. Possibly cross-referencing it against google analytics landing pages or querying the google cache for that page (SEOtools for Excel from Biels Bosma is good for this).
Similarly, if you have a definitive list of the URLs on site you could start with that list and query which are cached.
Harder than it seems isn't it? Hopefully one of those methods will put you on the right track.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I batch export top 5 or 10 SERPS for each keyword in my campaign
I'm a newby. I have about 70 keywords and I want to create a list of all of the websites ranking in the top say 10 positions. This is to create a list of competitors in my client's space. How do I do this? I poked around but can only see single-keyword search capability to get the full serp listing (top 50). thanks.
Moz Pro | | empiricalSEO0 -
8 point decrease in DA, increase in search traffic- should I disavow links?
Would a decrease in DA reflect the need to disavow junk links? My branded keywords recently bumped down two spots, whereas the keywords which have been bringing me the most traffic have been unaffected. Site: Edwardsturm.com DA decrease of 8 over past 3 months. Keyword "Edward Sturm" recently moved to second spot.
Moz Pro | | Edward_Sturm0 -
Google has a penalty on my website? How to resolve?
I recently purchases Moz pro, hoping to find out why my website isn't ranking on specific keywords. I'm done analytics on myself and my competition and it seems that I rank better than them on 95% of the categories (using OpenSiteExplorer and keyword research tools). So I purchases Moz Pro hoping I'd find the glaring error in my ways. I found a few small issues I was having, and have dealt with them. Overall however, they were small issues, and have led me to believe Google as a penalty in place for my website (SEO friend have also suspected this). In my Google Webmaster tools, nothing is marked under Manual Actions. So does anyone have a way to contact Google and have them directly look at my website to insure there is no errors in their system?
Moz Pro | | ZSuttonPhoto10 -
How do you run ranks now for clients that aren't in your local area now that Google customizes search results based on location?
How do you run ranks now for clients that aren't in your local area now that Google customizes search results based on location? What tool/service have you found that gives accurate rank results? To me it seems like there is no guaranteed global #1position any more, or is this the situation for only certain types of searches? Likewise how can anyone tool give a global accurate rank score?
Moz Pro | | greengroup0 -
Problem with our connection to your Google Analytics
It is the second time in less then a month when i get that error message. The first time I just removed the connection and re-added it. Is there any other way to fix it? is it the problem on my end or is it on SEOMOZ side?
Moz Pro | | SirMax0 -
Why did the crawl last night not show the same results i see in google?
Last night my keywords were crawled and it shows me that a key word is ranked 14. For 3 days now it has been rank 4 or 5. Is there a reason this is not accurate? I have not checked the rest of my keywords so i am not sure about those. Thanks
Moz Pro | | tom14cat140 -
How to get rid of the message "Search Engine blocked by robots.txt"
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest. Regards, Prashakth Kamath
Moz Pro | | 1prashakth0 -
Why do I see different ranking results for Bing and Yahoo?
If Bing is running the Yahoo search results then it seems to me the ranks for both Bing and Yahoo should be the same. So why are my SEOmoz ranking reports showing different positions for each?
Moz Pro | | Thos0032