Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Capital - Antitrust Conspiracy
I think we all have heard about Thumbtack breaking the rules w/ badges. Getting deindexed, then getting a 100M injection from Google capital and having the penalties removed: https://techcrunch.com/2014/08/20/service-marketplace-thumbtack-raises-100m-round-led-by-google-capital/ Our primary competitor is a different marketplace backed by Google Capital. Does anyone know of any low frequency products (reliant on SEO) backed Google Capital that has not won out within search? (i.e. is there any hope of competing against a low frequency marketplace after they have Google Capital backing?)
Search Behavior | | MarketGrowth0 -
Skewed results in Google
Im just curious.. Im sure a lot of you, like me, Google your own search terms on a fairly frequent basis. Does doing this skew the result you are actually seeing? As in, does Google tailor the results more to my liking, and therefore perhaps show my own site (which I would click through to often enough) higher on the page that it actually is? Sometimes I swear that happens. Steve.
Search Behavior | | blitzna100 -
GWMT - "Tag Site For Child Directed Treatment" Effect On Search / Rankings?
Hi All, We have a client who has been directed to tag their site for "Child Directed Treatment" in Webmaster tools to comply with AdExchange policies. The site is, generally speaking, directed at those between the ages of 13 and 16 along with their parents, but does NOT collect any data (No sign in, login, signups etc). You can find out more about the specific tag here (unfortunately not much more about it) https://support.google.com/webmasters/answer/3221080?hl=en Our concern is that we have never heard of this specific tag, and our client is asking us to let them know if that this will have no effect on search traffic or ranking. I can't find much in the way of anyone who HAS implemented this tag and the effects it has had on their site. They are ad supported, receive millions of unique hits a month, and the majority of their traffic is from branded keywords. Would love to hear from anyone with ANY experience or thoughts on this process and what to be aware of. Your assistance is muchly appreciated.
Search Behavior | | SearchMarketers0 -
Google De-Indexed Our SIte for Branded Terms?
Hello all, As of 10am Pacific on September 12th, 2013, my team has noticed that our site, www.wirelessemporium.com, does not show up on the first 5 or 6 pages of SERPs for branded terms like "wireless emporium." We have not received any messages from Google via Webmaster Tools regarding this. Major activity that we've been doing to our site is updating content, meta tags, and h1 tags, along with removing/301 redirecting certain pages that did not meet Google quality guidelines. We've also been purging our backlink portfolio of toxic links and URLs, both manually and through the disavow tool. No blackhat has been done to this site for a very long time (more than 8 months now). One thing to note is that we did have a manual spam penalty placed on us back in July of 2012, it expired in early August of 2013 after a reconsideration request was submitted, and a 2nd manual spam penalty was placed on us again later that month. We are submitting a 2nd reconsideration request this Monday. Could this or the recent Panda update have anything to with this? We are very much in need of opinions as to why this is happening to our site. 5adbd14a31de3a78b998df94f0b6d2be
Search Behavior | | eugeneku0 -
Using Google Analytics to See What Time of Day Visitors View My Site
Hi folks, My company has Google Analytics setup for all of our websites, but I am a bit stumped on something. Now, this may not be possible, but am I able to see what time of day visitors most frequently view my blog? I would like to optimize blog post publishing for when I know we have in influx of visitors, yet I cannot find this information on GA. Any input would be much appreciated. Regards,
Search Behavior | | Instabill
Meghan0 -
Huge difference in search results depending on the country?
I have some keywords ranking in pages 2 and 3 in the US but for instance, in Canada, the same keywords rank somewhere between pages 15 and 25. When I check my US rankings, I use a Windows VPS located in the US. When I check my rankings in Canada, I use a computer physically located in Canada. Basically, about 80% of my rankings have gone away for countries other than the US. I have not set any geographical target for my website in GWT. How come there is such a big difference in search results across different countries?
Search Behavior | | sbrault740 -
Google Instant Search? REALLY?! Why is this the result???
First, this is one I thought I would never ask, but: Could this be true? I had noticed my MacBook Pro was eating some power and wanted to see if I could optimize my settings so to take advantage what battery life was available. (I swear that is the truth). So, I am at a Starbucks on someone and typed into Google search bar the following query - Best way to conserve - and, received the following instant search results before I could type - power on a macbook pro I am not putting it here as I want you to ask yourself what came up. I have attached a screenshot. My colleague arrived and without prompting she retried and got the same. So I am curious: Do you get the same result and why do you think that result is so predominant? This is funny. efuts.png
Search Behavior | | RobertFisher2 -
Can we rank as a related search query on a competitor's brand keyword?
One of my clients wanted to know whether it's possible for him to rank as one of the related pages on Google for a brand name of a competitor. I honestly don't know whether it's possible so any ideas whether it's possible and if so, ideas on what can be done is greatly welcome. Thank you in advance. 🙂
Search Behavior | | als0070