Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google not giving ranking to the intended page of my website.
Hello friends, This is my very first question, I hope I will explain my issue correctly. I have created two pages related to SSC CGL keyword on my website: https://www.ibtindia.com/ssc-cgl-notification-exam-date 2) https://www.ibtindia.com/ssc-cgl-apply-online I want to target the keyword SSC CGL Apply Online on the 2nd URL but Google is only considering the 1st page for all the "apply" related keywords as well. Can anyone suggest to me how to get the second page in rankings for its intended keywords? I will be very thankful for this help. I tried adding image but its showing dummy. Please you can check it op GOOGLE SERP Second page
Search Behavior | | namitathakur0 -
Google Analytics Tagging
Hi. I'm trying to figure out a solution to three questions one of my current clients has asked me in regards to Google Analytics tagging, and I'm unsure how to respond. Can anyone help? See below the questions, 1. In Google Acquisition > Overview, their paid media is reporting as "Other". They do not run any Google paid ads. They only run Facebook paid ads. Is there a way to update the source so that it says "Paid" versus "Other" within the default channel? The current solution was advised to create a channel group that the client has to then tick on overtime they want to see this data with the correct labeling. They would prefer to see it in the default. Is it just a matter of going into the *default channel, choosing the "Paid" option, and then specifying the source/medium that contains Facebook, CPC, or referral to be categorized under this channel? Or is it something else? *Aware that changes to the Default Channel are permanent changes and will change how new traffic is classified. 2. In Google Acquisition > Overview > Referral, the clients website is showing up as a referring domain, both the TLD and the subdomain. My understanding is that it should actually be reporting under the "Direct" channel. How do I correct this? Is it just a matter of updating the Direct channel to include those domains? Or do I need to update the settings? The domain's www. http: all 301 redirect to their https://domain.com and https://subdomain.domain.com. Within settings it has been specified as www.domain.com and URL is http:// - also noticed that Bot Filtering has not been checked, assuming this could mess up the analytic data if not define? Do you know? 3. Audience segmentation > The client wants to be able to define it's audience by shopping intent and informational intent. Is there a clear way to do this, for example, by keywords used, e.g. buy, product name, entry (shopping intent), versus e.g. non-purchase intent, entry to the blog, length of time on site (info intent). Would be happy to have a conversation about the last question, since I'm conscious that there are probably multiple ways to define this - thanks. To the group, thank you for readying my questions and helping me with these solutions - your time is appreciated and valued. Sincerely, Amanda
Search Behavior | | AmandaValle.Digital0 -
Google Analytics Segments - Can't find location
Hi all, I am trying to set up a Google Analytics segment with the location as 'Kurdistan'. I have tried all location groupings (city, region etc) and am unable to locate it. Anyone else not able to locate it? Or does anyone know why I can't? Thanks,
Search Behavior | | Davinia22
Davinia0 -
Industry Benchmarks for Google Analytics
Hey Mozites! I was wondering if anyone has come across any good resources for industry benchmarks relating to google analytics? (Social stats would be great as well, but Google Analytics is my main concern) Specifically, I have a client that is a bank and I would love to be able to compare their stats (traffic, unique visitors, bounce rate, time on site, conversion rates...) to other regional banks. Any ideas? Thanks for reading and for your help!
Search Behavior | | iBecC0 -
Is it possible to know if visitor arrived at the web page via organic search and if so, show some content?
Hello, Is it possible to know if visitors are arriving at a web page via organic search? Background: We have a section of job description pages to explain typical tasks. These have very high bounce rate (some 100%), and I think people are confusing them with actual jobs. For example "stage designer". Many of those keyword we have very high rankings. I am thinking of having a small notice at the top of those page to say something like "if you are stage designer job, check out our job section". Thanks
Search Behavior | | CreativeChoices0 -
Google Webmaster Tools notice of detected unnatural links to our website
Hi, We have been using SEOmoz for a good few months now and have found it incredibly useful. Unfortunately however I think we may have slipped up a little bit as we have just received the following message in our Google Webmaster Tools: Google Webmaster Tools notice of detected unnatural links to http://www.refreshcartridges.co.uk/ I have a funny feeling I might know what the problem is but I'd like a confirmation before I potentially go off on a wild goose chase. We also own the domain name www.computerarticles.co.uk and recently started introducing the author at the bottom of the post along with both a link to the website homepage and a link to a deep, popular page on our website. For example, check out http://www.computerarticles.co.uk/twitter-time-saving-tools/ We genuinely didn't realise this was a bad thing and thought it would just provide Google with a way of deep linking straight in to popular areas of our site. I would really appreciate advice as to whether you think it might be these links that are causing the problem or whether there is something else that could be causing the problem. In the event it is these links that are causing the issue would you recommend removing the entire 'about the author' page (as we have published around 500 pages on that site) and simply put a link in the blog roll or simplify it to just link to the homepage. I appreciate in advance any help you could give. Regards Chris
Search Behavior | | ChrisHolgate0 -
Google Query Contamination.... Are you seeing this??
I have been searching on Google.com with FireFox this morning. I searched for a state name.... like "georgia". Then I search for a product... like "guitars". For the Guitars query the first page of the SERPs include some music stores in georgia. So, Google is contaminating your search results with information from your most recent query. Are you seeing this too? Has Google been doing this for a long time? Or, I am going crazy?
Search Behavior | | EGOL3 -
Google Rel="Next" & Rel="Prev"
Hello, I have a catalogue website and I am implementing the rel="next" and rel="prev" to the website. My question is that we do have a view all page also, which apparently Google likes over a 'page1'.. Should I add the canonical to this page? I already have it set to WEBSITEURL/sonos (which is going well) I don’t want to have to change this to [URL]sonos/view-all (which is my view all link) as the first page is getting ranked well I am then telling Google no, the view all page is the parent. Any advice would be very much appreciated. Thanks Rick
Search Behavior | | Lantec0