What's a better way than this to learn about web crawler?

Doing a bit of background research on Brewster Khale (creator of Internet Archive), I came to know this idea of archiving websites all over the world was pretty common back in the days. Main core of archival system is WARC format (web archive format), they sidestep inode busting (explosion of nodes in filesystem when you store too many individual directories / files). WARC contains 8 record types: warcinfo, response, resource, request, metadata, revisit, conversion and continuation. The records are sequentially placed next to each other - repeated many times. The whole warc is gzipped, but to support random access, each record inside this container format is individually gzipped, hmm clever (tradeoff is +10% in overall size).

No restrictions on record count and size, picking a sweet spot is your headache. warc stores image bytes, both request & response bodies along with the plain html body. They say just saving html body is too lossy. Reading is through scanning sequentially and unwanted records are skipped by parsing already recorded Content-Length header. Wait - they have a index over this setup too (CDX), storing URL-file association, offsets & length recorded for faster disk seek over a range of bytes.

Internet Archive’s petabox is passively cooled, thanks to geography.

Huhh… I haven’t even gone into the sweet crawler part. Please read that yourself, many thanks!!: https://hackernoon.com/the-long-now-of-the-web-inside-the-internet-archives-fight-against-forgetting (and more about legal battles, decentralized storage, and policy on AI datasets and training).

crawler source code: https://github.com/internetarchive/heritrix3