Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Follow the wormhole through a path of communities !webdev@programming.dev
Do I understand correctly that this has all the building blocks which make up various lossless compression schemes (dictionaries, run length encoding, delta encoding, etc) and kit bashes them together for your specific dataset?
I was under the impression that modern file formats and compression schemes already kind of do that as part of the normal compression process, so I was a little surprised by how much they beat some of the competition in their benchmarks.
It can only do the impressive numbers after you give it a schema for the data, and the data has to be structured in a way that follows a rigid schema, so it's not magic.
OpenZL compresses data into a self-describing wire format, any configuration of which can be decompressed by a universal decoder. OpenZL’s design enables rapid development of tailored compressors with minimal code; its universal decoder eliminates deployment lag; and its investment in a well-vetted standard component library minimizes security risks.
The freedom afforded by OpenZL’s graph model is a double-edged sword: the capability it offers comes at the cost of a combinatorial explosion of choices that must be made—namely which components to use and how to compose and configure them. There is no one-size-fits-all compressor
Webassembly for compression or something, but it's made by the zstd people so maybe it will turn out to be magic eventually.
On one hand the x2.06 ratio for the SAO data is extremely good, zpaq and paq8px are 50x and 10000x slower at encoding and decoding and still can't even reach that ratio (I didn't try even more extreme compression, paq8px -8 already took 20 minutes).
On the other hand,
enwik: A Case Study Where OpenZL Performs Poorly
We present this case study to show that OpenZL is not a magic bullet for all use cases. The shortcomings are twofold. First, the standard codec library can achieve good results for many datasets, but is not currently optimized for human text. Second, English text is a highly compressible format for which domain- specific transformations are especially useful. xwrt combines multiple such transformations to achieve its results. With this in mind, we are optimistic that porting text-specific codecs and text parsing will improve the performance of OpenZL compressors in this domain.
