Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ways to Identify Popular Search Terms
Certain searches seem intuitively like they would be popular, but don't appear so in my keyword research on Moz Pro. For example, I am a therapist and would have guessed that a lot of people would be searching for "online therapy California" during this pandemic, but actually those terms are not popular. I looked at Google Trends to see if I could understand this better, but It wasn't very helpful. Any other suggestions for where to get more information when search terms you would expect to be high volume don't appear to be so?
Search Behavior | | LPantell0 -
Google Index Issue - Indexing pages that don't exhist
Hi All, I have noticed a weird issue when performing a search on Google to show me all the pages it is indexing of our site. site:www.one2create.co.uk It brings up most of our website pages but then is also brings up a few HTTPS urls (our site has not been converted to HTTPS yet) but also the URL path, Title, and Meta Description are from one of our clients websites (an Automotive Job site). When clicked they take you to a generic 404 server error page, not our branded 404 page. The site that it has taken the url, title and meta description from is on a different server completely so I don't see how it has even managed to get that information and linked it to our site? Has anyone seen anything like this before? And what is the best way to fix it? We have asked Google to re-index the site but still no luck.
Search Behavior | | Jvickery0 -
What is the best way to learn google analytics?
Hi I'm trying to get my head around google analytics as a first time user. Can any suggest the best place to start after I have taken a look through the whole thing, how do I go about managing and tracking performances on my website? eg: Registrations/sign ups, conversions all that sort of thing Some advice will be much appreciated Thanks
Search Behavior | | edward-may0 -
Store locator searches online and offline store visits
Hi All, What is the accepted % for visitors that conduct a store locator search online actually visiting that particular store for the nutrition industry? (For example if 50 people used the store locator search on the website , we can confidently say that 25 of them actually went to that store) Any input or reference to studies/white papers on this topic would be appreciated. Cheers, SEO5..
Search Behavior | | SEO5Team0 -
Google keyword planner shows low search volumes
Hello, There are some terms that you'd expect to see a lot of results. In fact, in the old days of Keyword Tool, I recall seeing thousands of results for a keyword like "anderson cooper". Today, I see a small amount as a monthly average. Am I doing something wrong? All I really want to do is find search volumes on particular keywords either globally or local to a country. And it's proving soo hard! 😞 Screen_Shot_2013_11_26_at_11_13_20_PM.png
Search Behavior | | mhamilton0 -
Would you instinctively mistrust a company that has the top 5 or more search listings?
I have seen a few "lucky"(?) companies dominate spots 1-5 of search results. When I see that, I generally go to #6, 7, or even 10 before clicking. My gut reaction is that the company with 1-5 has manipulated content just to pull me in - much like the guys on the streets of NYC or Vegas shoving handbills at you. Is there any evidence/research/poll about the consumer/user reaction when this occurs?
Search Behavior | | AMSVansSEOTeam0 -
Pro/Cons of JUST using Google Analytics
I find myself in a situation where I am having to defend why purely relying on just Google Analytics for site data i.e. exit pages, keywords, bounce rate etc... is not good practise. So I thought/wondered if other people have come up against this. What are the Pro and Cons for ONLY using Google Analytics for your site data....
Search Behavior | | JohnW-UK0 -
Google button +1 ( SEO point-of-view )
Hello Folks, First of all, I want to start by: this is a "discussion" topic, which theres no right or wrong answer. I would love to hear from everyone your toughts about the GOOGLE +1 BUTTON , for the SEO point-of-view indeed. I assume that I did not add it to my websites yet, theres some reasons why. You guys added it? How is your audience behaviors against the button, they often click on it? It changed anything in your ranks, after they clicked on it? Thanks in advance.
Search Behavior | | augustos0
