Programming

27603 readers

95 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 3 years ago

MODERATORS

snowe@programming.dev

Ategon@programming.dev

UlrikHD@programming.dev

bugsmith@programming.dev

Spyro@programming.dev

OpenZL delivers high compression ratios while preserving high speed, a level of performance that is out of reach for generic compressors. (openzl.org)

submitted 1 week ago by cm0002@infosec.pub to c/programming@programming.dev

2 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] Starfighter@discuss.tchncs.de 3 points 1 week ago* (last edited 1 week ago) (1 children)

Do I understand correctly that this has all the building blocks which make up various lossless compression schemes (dictionaries, run length encoding, delta encoding, etc) and kit bashes them together for your specific dataset?

I was under the impression that modern file formats and compression schemes already kind of do that as part of the normal compression process, so I was a little surprised by how much they beat some of the competition in their benchmarks.

[–] sus@programming.dev 3 points 1 week ago* (last edited 1 week ago)

It can only do the impressive numbers after you give it a schema for the data, and the data has to be structured in a way that follows a rigid schema, so it's not magic.

OpenZL compresses data into a self-describing wire format, any configuration of which can be decompressed by a universal decoder. OpenZL’s design enables rapid development of tailored compressors with minimal code; its universal decoder eliminates deployment lag; and its investment in a well-vetted standard component library minimizes security risks.

The freedom afforded by OpenZL’s graph model is a double-edged sword: the capability it offers comes at the cost of a combinatorial explosion of choices that must be made—namely which components to use and how to compose and configure them. There is no one-size-fits-all compressor

Webassembly for compression or something, but it's made by the zstd people so maybe it will turn out to be magic eventually.

On one hand the x2.06 ratio for the SAO data is extremely good, zpaq and paq8px are 50x and 10000x slower at encoding and decoding and still can't even reach that ratio (I didn't try even more extreme compression, paq8px -8 already took 20 minutes).

On the other hand,

enwik: A Case Study Where OpenZL Performs Poorly

We present this case study to show that OpenZL is not a magic bullet for all use cases. The shortcomings are twofold. First, the standard codec library can achieve good results for many datasets, but is not currently optimized for human text. Second, English text is a highly compressible format for which domain- specific transformations are especially useful. xwrt combines multiple such transformations to achieve its results. With this in mind, we are optimistic that porting text-specific codecs and text parsing will improve the performance of OpenZL compressors in this domain.