Exporting Google and Bing Search Results
-
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages.
Cheers
-
There is a chrome extension called "scrape similar" that is useful for doing small batches of stuff like this. However it does have a couple of limitations in that you have to view each page & google will not show you all pages of a large domain. However it is quite easy and effective for sites with under 1000 pages.
https://chrome.google.com/webstore/detail/mbigbapnjcgaffohmbkdlecaccepngjd
The process can be sped up using other tools. I use tool that is designed for black hat forum/comment spamming to do SERP scrapes like that. Even if I did such spamming (I don't), I don't actually think this is a very good tool to do it with. However it is rather good at scraping results from google. However, again you are limited to how many results Google/Bing choose to show you.
If you need a bigger list then log files might be the way to do. You can get a list of all crawled URLs for any particular agent (including the likes of googlebot) from your server logs. Some hosts limit the size of these, so it might be worth checking before you start. However the data does get collected. The downside here of course is that you need access to the logs.
Of course crawled is not the same as indexed. Once you have that list you might need a further step to see which is indexed. Possibly cross-referencing it against google analytics landing pages or querying the google cache for that page (SEOtools for Excel from Biels Bosma is good for this).
Similarly, if you have a definitive list of the URLs on site you could start with that list and query which are cached.
Harder than it seems isn't it? Hopefully one of those methods will put you on the right track.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to add my company in google search search result in bangalore (INDIA)
Hello, i want to add bangalore which is a district in INDIA to moz so that people searching from that bangalore would find our company listed in google search .Happens to be that i am not able to enter any locations apart from USA. It would be of great help if you could assist us in doing the same.Also wanted to know how how to target preferred locations in India. Thank you
Moz Pro | | Jabez0 -
Does SEOmoz have a Keyword Research tool similar to, say, the Google AdWords tool or the WebCEO Keyword Research Tool? And where might that be? (Sorry, I'm very new to SEOmoz Pro.)
I'm looking for an SEOmoz version of the classic WebCEO Keyword Research that would give you effective suggestions based on a keyword inquiry. I've made the switch from WebCEO, but I'm trying to find something similar to that Keyword Research tool. Am I going to just need to use the Google AdWords tool for this function or does SEOmoz have it's own version?
Moz Pro | | SmokewagonKen0 -
My Google Keyword Rankings are Missing. Anyone else having this problem?
I logged in to pull my reports for my clients and noticed that I am missing all of my ranked keywords for my campaigns for the Google columns. One word is ranked. All the rest have a TUES sign that looks like it references Tuesday. Is anyone else having this issue? I had over 30 words ranked last week. Only showing 1 today. The rankings for Bing and Yahoo are intact though. There is no report showing a decline in ranking for all those words either (in the google column.) This is also happening in another campaign of mine but they have a WEDS in place of the google ranking... ugh Frustrated...
Moz Pro | | JChronicle1 -
SEOmoz Keyword Ranking - Does it work for local search?
I was just wondering how SEOmoz calculates keyword rankings up and down every week. Shouldn't there be a way to have SEOmoz calculate rankings on a Local and Non-Local Basis? For example I have "Local Dentist" as a keyword in SEOmo for my dental client. SEOmoz shows that we are not in the Top 50. | Local dentist Keyword History Ranking Analysis | Not in Top 50 | Moved unknown 0 spaces | Not in Top 50 | Moved unknown 0 spaces | Not in Top 5 | If I change my location in Google browser to the zip of where the office is located, we
Moz Pro | | Czubmeister
show up in position 14(for the home page) & 15 (for an inside page) How can I show the keyword rankings to my client if the keyword ranking report doesn't reflect searches done in my clients local area?0 -
Problem with our connection to your Google Analytics
It is the second time in less then a month when i get that error message. The first time I just removed the connection and re-added it. Is there any other way to fix it? is it the problem on my end or is it on SEOMOZ side?
Moz Pro | | SirMax0 -
How does moztrust compare with google page rank?
I have a trial Seomoz membership. I've been comparing my site to some of the competitor websites and Im confused about the moztrust vs google page rank. For instance some of my competitors have higher moztrust but my site has a higher google page rank. Isn't moztrust meant to duplicate google pagerank?? Why would these vary so much?
Moz Pro | | blac67890 -
Confused by Google Mobile App (on Blackberry) results??!
First off, Hi guys I'm a new user here, in fact only in my second week of my trial period. However, I can assure you that I'll be continuing my subscription as this website is 'one hell of a bit of kit!'. Now, to my predicament. I have a website: http://www.limegreenofficeproducts.co.uk which I am trying to move on up the rankings in Google (just like everyone else...). Well, I have followed the instructions and guidance through the Campaign Manager and I have 'A' ratings now for a couple of my preferred keywords, namely 'Office Supplies' & 'Office Products'. I also have a number of textlinks with these exact terms, some quite powerful (I'm the only outbound link on a Homepage PR5 on one). Anyway, being a complete and utter control freak - I wake up in the morning and check my rankings using the Google Mobile App for Blackberry whilst throwing as much coffee as possible down my neck. Basically (if you're not familiar with this app, it is just the same as connecting to the mobile internet and carrying out a search - or at least it should be). Well I was really excited to find that I was ranking at No.41 for 'Office Supplies' and No.17 for 'Office Products'. When I fully woke up and ventured to the office, I checked on the Mac through the normal Google UK and I'm nowhere, for either? What makes it even more confusing is that the results on the mobile seem to be intermittent - so if I check at 11.00am I'm No.17, 11.05 I'm nowhere, 11.10 back to No.17 - but only on the Mobile App. I have the Mobile App set up to Google UK, so that can't be the problem. I'm just wondering if either the Mobile App is ahead of the 'Real' Google UK results, or behind.The main reason for asking, is so that I can establish whether what I am doing is having a positive, or negative effect on the rankings. And if this is an quicker way to find out - then great! I assume the advice to come back will be '..ignore the mobile app..' but as it's being kinder to me than the 'Real' Google I'd like to be a bit kinder to it, and give the little fella the benefit of the doubt. But having said that I just checked the search results (Top 1000) for Keywords 'Office Supplies' & 'Office Products' - For Office Products the site was No.614 and for 'Office Supplies it wasn't in the top 1000, ouch. I know these things take time, as I have worked on a couple of other sites of ours and it seems that as soon as you are about to throw the towel in, the results just kick in. I'm not expecting miracles overnight, far from it - but it has me really confused. Does anyone have any suggestions/advice?(except '...get a life coffee fiend') Regards Limegreen
Moz Pro | | Limegreen0 -
Why can I only add three search engines to a campaign?
I need to track global keywords rankings for a web based product. Only being able to add three search engines to a campaign is limiting. I need to track at least 5 regional versions of Google, and curently have to duplicate my campaign to do this. would prefer to keep it under one campaign.
Moz Pro | | Vend0