Exporting Google and Bing Search Results
-
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages.
Cheers
-
There is a chrome extension called "scrape similar" that is useful for doing small batches of stuff like this. However it does have a couple of limitations in that you have to view each page & google will not show you all pages of a large domain. However it is quite easy and effective for sites with under 1000 pages.
https://chrome.google.com/webstore/detail/mbigbapnjcgaffohmbkdlecaccepngjd
The process can be sped up using other tools. I use tool that is designed for black hat forum/comment spamming to do SERP scrapes like that. Even if I did such spamming (I don't), I don't actually think this is a very good tool to do it with. However it is rather good at scraping results from google. However, again you are limited to how many results Google/Bing choose to show you.
If you need a bigger list then log files might be the way to do. You can get a list of all crawled URLs for any particular agent (including the likes of googlebot) from your server logs. Some hosts limit the size of these, so it might be worth checking before you start. However the data does get collected. The downside here of course is that you need access to the logs.
Of course crawled is not the same as indexed. Once you have that list you might need a further step to see which is indexed. Possibly cross-referencing it against google analytics landing pages or querying the google cache for that page (SEOtools for Excel from Biels Bosma is good for this).
Similarly, if you have a definitive list of the URLs on site you could start with that list and query which are cached.
Harder than it seems isn't it? Hopefully one of those methods will put you on the right track.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Google really care about cheaters?
In analysing my competitors website I have discovered that they have over 600 fake pages that appear to have been created by some sort of internal search engine software, all pages are exactly alike except for the keyword and the words "no results found" on every page. I have discovered, elsewhere, that their duplicate content is abnormally high, and that their pages are stuffed with keywords.(in on page grader) I come out top in every section in Moz open site explorer, except internal links and total links. Yet they sit at number 1, while my site is number 2, I believe that copying their antics would surely make me fall foul of Google, and certainly would never risk it. but with the keyword only getting 30 searches a month, are google bothered about them doing this, should I resign myself to not being number 1 and is there anyone on this Forum who believes that I should use the same tactics, to get above them.
Moz Pro | | jefftracey1 -
Can I setup SEOmoz PRO's 3 organic traffic metrics within Google Analytics (as custom reports)?
I like the 3 organic traffic metrics in SEOmoz PRO: 1) Organic Search Visits
Moz Pro | | AndreVanKets
2) URLs Receiving Entrances Via Search
3) Non-Paid Keywords Sending Search Visits They rock - especially for a high-level view of SEO / Organic performance. Is it possible to setup these metrics as custom reports directly in Analytics? Does anyone have these reports setup? Or maybe there are similar custom reports that give a similar high-level view?0 -
What's up with Yahoo and Bing in my keyword rankings?
SEOmoz is good I've learned a lot from it. I'm on my 3rd month and now have 8 keywords in top 50 for one site and 1 keyword in top 50 for another. This is for Google US. For Bing and Yahoo! US however I am still not in the top 50 for anything. I have read elsewhere that meta-tag page words can be regarded by these engines as a spam signal. In my meta I have used competative keywords that are relevant to my content, some of which are in the body, but are not my actual targeted keywords as a red-herring to potential competitors. Also might be usefull for other smaller places that still use meta-tags. Could not appearing in BingHoo! have something to do with their alogrithms and the meta-tags? or is it an SEOmoz issue? I have another question also. The " / " after a domain name doesn't appear in my Firefox address bar (I think I did a 301 redirect), but when I copy and paste it into an entry box (e.g. for a link submission) it puts the " / " there at the end which I take off. Does this make any difference?
Moz Pro | | Zoolander0 -
My Website is not being crawled by Google last 15 days
Hello, Required emergency help, one of client website http://www.mediaxpress.net/ is not being crawled since last 15 days, What can be the reason. I have Checked Google Webmaster Tools, it not showing an Issue and even SEOmoz crawler is crawling it normally. I have updated some data on pages, and i want to get it crawled by Google. But Google is not crawling in anyway.
Moz Pro | | CommercePundit0 -
UK Google Results .com V's .co.uk
In the UK lots of people use google.com rather than google.co.uk. See: http://weblogs.hitwise.com/robin-goad/2008/04/how_popular_is_googles_pages_from__the_uk_search_option.html The above data is a bit out of date but still I know lots of UK people who search on google.com. When I depersonalise the results and search there is a difference between what I see on google.com V's what I see on google.co.uk. Indeed one of the companies I work for ranks ok on google.co.uk, but not so well on google.com Questions: How would I best track ranking for UK based users who use google.com AND/OR google.co.uk using the SEOmoz tools (or other tools if appropriate)? I note that in "Google Webmaster Tools" you can change the setings to "Target users in United Kingdom". Given the fact that UK users use both google.com AND/OR google.co.uk would it be better to change this to unlisted or deselect the "Target users in" option? Thanks in advance. Justin
Moz Pro | | GrouchyKids0 -
How to rank each page correctly in the search engines
Hello everyone, I am new to SEOmoz and SEO in general. I am trying to wrap my head around everything and one of the issues I am having trouble grasping is how each individual page on our website plays into ranking for certain keywords. I will explain in more detail below: 1.) We are not a blog, we have a pretty static website. 2.) We can add as many pages to our website as needed (for SEO purposes) 3.) Our site is fairly small, about 25 pages on the website. Now with that information above, here are our main questions & concerns: 1.) What is the best way to rank in the search engines for 50 of our best converting keywords with only 25 pages on our website? 2.) We have 2-3 competitors that rank in positions 1-3 for about 30 of our keywords we are trying to rank for. But no matter what you type in, for example, "cars" "buy cars" "buy a car online" "truck prices" etc, the results that the search engines display our their homepages. Not the other pages on their websites that actually rank for each one of those individual keywords. Is this normal? That is something that we would like for our rankings to do if we have any control over it. 3.) Should each page try to rank for a unique individual keyword? For example, when using the SEOmoz On-page Optimization tool, should I try ranking 2 pages for "buy cars online" or just 1 page? Say my homepage has an "A" grade for the keyword, "buy cars online" but so does my second page on my website. Are these pages essentially competing with each other? Or will the search engines pick their favorite page and display the one they like the best? Any help is greatly appreciated and I look forward to getting this cleared up. Thanks
Moz Pro | | threebiz0 -
How come there are no links to my website according to SEOmoz Competive domain analysis, while in google webmaster i do see links.?
I dont see any links to at all when i do a Competitive Domain Analysis in SEOmoz. However i do see links in google webmaster tools. this strikes me as odd. Also when i use open site exployer my website dont seem te be found. In google im on page 9 on my focus keyword so i do think there are links to my site. I would like to know what i can do so i can analyse my links in seomoz Competitive domain analysis. Many thanks. url: http://www.sadpanda.nl
Moz Pro | | Aquive0