Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best way to learn google analytics?
Hi I'm trying to get my head around google analytics as a first time user. Can any suggest the best place to start after I have taken a look through the whole thing, how do I go about managing and tracking performances on my website? eg: Registrations/sign ups, conversions all that sort of thing Some advice will be much appreciated Thanks
Search Behavior | | edward-may0 -
Personalised Geo-targeted results - How does Google pass link juice?
Hello, Many websites now serve specific home page offers based on the location of the customer, my question is, how does link juice flow around a site when the links (this case from the homepage) are served up based on a visitors location? Internal links from your homepage are valuable for ranking that product well in the SERPs so how does Google deal with this? So, for example, a car hire website based in the UK. If you arrive on the care hire website sat in Manchester (Northern UK city), on the homepage the website serves offers of car hire deals in Manchester, Leeds, London and international destinations. If you arrived on this website from London (Southern UK City), you would not see the Manchester link at all but London, and other cities in the South. In this case, when Google crawls the car hire website, it will see internal links but a)which version and b) is there any way of sharing this link value around? Basically, we want to understand if Manchester in this case will get the benefit of an internal homepage link from Google even though we only show Manchester to people FROM Manchester, OR, do Google only give juice based on one version of the website, a generic UK version? Or to put it another way, is there any way of cashing in on both geo-targetting the customer based on their location AND getting link juice from those geo-specific home page links? Perhaps there is some code or way of telling Google that people from Manchester (a certain % of our visitors) will see a homepage internal link for Manchester that will pass some small % link value?
Search Behavior | | xoffie0 -
Banned from Google
Hello experts I need some help. I have a customer that have been banned from Google´s top ranking. I need an expert to get the cusotmer back on the track again.. Cann you help? Regards Peter
Search Behavior | | seopeter290 -
My website disappears off google!
So this might be kinda of a weird question... Every morning and night I check the ranking of a website that I am building.. The ranking has gone up a lot the last two months. It went from the fifth page to now the second page. I have a issue where some days I check Google my website is completely gone! I go through every page for my keyword and it's not there! After a couple of days of frustration I check again and all of a sudden it is there but now at a higher ranking... I went through the code to make sure there's a not a not follow code in the robots.txt page... Btw another weird thing is so then I look up my website on a google out of country like google.sg and I'm ranking first page like number 5 but again disappeared off google usa. Literally driving my crazy.. does anyone know why this could be? Btw the first time it disappeared I went into webmasters and sent a request because I thought I got penalized but they responded they could not find any spam and I was NOT penalized...
Search Behavior | | BecCan0 -
Google ranks our competitor above us on 1000's of branded queires!!!
Hi all, I have noticed a very bizarre phenomenon in Google SERPs. When I search for a branded keyworks [Product + our brand].
Search Behavior | | ref.price
Amazon.fr appear above us on thousands of results. Google even ranks Amazon above us for queries like [ PriceMinister google plus]. I have tried to ask Google about it but I can’t seem to get an answer. Here is the topic I posted on Google’s forum: http://productforums.google.com/forum/#!category-topic/webmasters/crawling-indexing--ranking/DFvTPr14o_o This seems like a mistake on Google’s side, some kind of semantic association with our two brands! Basically they are sending our customers to our main competitor even though they specifically searched for our brand (PriceMinister). I find the phenomenon quite interesting for the SEO community and frustrating for our company. Does anyone have ideas on this one? Do you think it's a bug from Google? Cheers Oliver0 -
Massive Google Drop on two sites!
Hi All, We have experienced a massive Google drop on Two of our eCommerce websites in the past week. From Feb - July we were ranked 2 for various key words In Aug we dropped to 8 In September to 96 if not lower. We pay for monthly link building and add unique product info each week (but not much new content such as articles or blogs). My SEO guys has not been that helpful and just sent me a link to the Panda update blog, which doesn't mean a great deal to me. Obviously this has had a massive effect on business, and ideally i need to diagnose the problem and find a new startegy on moving forward, so most likely looking for a for a new SEO guy/company as well. I need someone is proactive, communicates well and make constant suggestions about moving forward not just link building. One thing I would like to add is these two site have a different homepage, category and range structure, but do share some if the 2000 product database. Can anyone help? Any advice on sorting this would be gratefully appreciated. Thanks M
Search Behavior | | etsgroup0 -
Irratic page rank results - why?
Hi, Hi- I've changed the title and reposted this question as I didn't get any insight. I really am going around in circles for this; so any help that can be given would be really appreciated.I know that you guys have the knowledge so really hoping you can share it! Thanks - background: I run two sites: www.ragsmetals.com and www.yourgoldfund.com: They offer similar products but to different markets: one is predominently trade, one predominently customer focused. They vie for the same keywords.e.g. Buy Gold My Issue: I have been trying to build www.yourgoldfund.com up from an SEO point of view; it is FALLING in the rankings; I am doing NOTHING with www.ragsmetals.com and it is rising in the rankings. I checked webmaster to see if there were any warnings as a result of penguin (found out how to do this on here - thanks for that!) there are none. I have optimized the pages a lot. I have put time into getting in bound traffic, and even more excitingly some experts (from reputable domains) have just linked to our site because they like our content! So to me things should be improving . NOTE: My OSE listed inbound links are incredibly diferent from google Webmaster. What can I do. I am a scientific person so I love the fact that I have 2 sites to compare...interested to hear your thoughts. Many thanks in advance
Search Behavior | | girlie0 -
How do you know if you have been penalized by search engines?
How do you know if you have been penalized by search engines? I logged into my Google and Bing webmaster accounts and all looks ok, but in the last 30 days, my organic traffic has decreased 33%. My traffic usually steadily increases and if there is ever a decrease, it's usually very small.
Search Behavior | | StreetwiseReports0