Welcome to the Q&A Forum

Only basic stuff: URL, Title, Description, and a few HTML elements.

I am aware that building a crawler would be fairly easy, but is there one out there that already does it without consuming too many resources?

I want to crawl a site that has so many pages that Xenu and Screaming Frog keep crashing at some point after 200,000 pages.

What tools will allow me to crawl a site with millions of pages without crashing?