Compressing a Histogram from 16GB to 2KB

Show HN: Compressing a Histogram from 16GB to 2KB

Based, Compressing a Histogram from 16GB to 2KB works in show hn launch.

Profile

Thought I'd dive into this cool streaming/approximation algorithms problem I encountered a few months ago.TLDR: we can create our own custom floating point representation for encoding and decoding integers and use this to index into a tiny 2D histogram, upper-bounding approximation error based on the number of bits per integer we specify to keep.---Broadly useful for aggregating statistics from massive data streams of user data. Also turns out this is incredibly similar to data structures used in production, like HDRHistogram.

Within the show hn launch space, the company focuses on its core offering rather than a broad product line.

Its current recorded stage is Show HN launch.

Website: alexkranias.com/essays/histogram.html