Xor data compression

“We addressed this by repurposing an existing XOR based floating point compression scheme to work in a streaming manner that allows us to compress time series to an average of 1.37 bytes per point, a 12x reduction in size.” ( source) .

Don’t send everything, just the parts that changed.
On the receive side, you simply reverse the XOR operation using the current state array.
Even if you use another compression algorithm, changing your data to be mostly zeros is going to help the compressor.

Additional reading:

1. Gorilla:
2. A Fast
3. Scalable
4. In-Memory Time Series Database One of the earliest lossless compression algorithms
5. Dictionary compression (in particular
6. LZ-based compression) is the ancestor of many compression schemes used today
7. including :
8. LZW (used in GIF) and DEFLATE (used in PNG
9. gzip )

Here we cover the second, XOR -based compression, which is something that typically applies to floats. (Note:

1. Developers will often refer to “Gorilla compression”
2. which is generally at least one
3. if not both
4. of these techniques

“We addressed this by repurposing an existing XOR based floating point compression scheme to work in a streaming manner that allows us to compress time series to an average of 1

37 bytes per point, a 12x reduction in size

XOR compression is a form of bitwise differential filter, it relies on searching of fragments of bit-strings to improve the compressibility of the data by the second stage of encoding (BitMagic uses Binary Interpolative Coding)

XOR is a binary function "exclusive or", the resulting bit evaluates to one if only exactly one of the bits is set

One of the important properties of XOR is that it is self-inverse

It means that if the result is XOR-ed with an argument it gives another argument