MinIO Server supports compressing objects to reduce disk usage. Objects are compressed on PUT before writing to disk, and uncompressed on GET before they are sent to the client. This makes the compression process transparent to client applications and services.
To enable data compression, use mc admin config set to set the compression key enable option to on . Replace ALIAS with the alias of a configured MinIO deployment.
Compression + Encryption at Rest
By default an extra parameter is required for MinIO to compress data to be encrypted on disk.
This is to ensure that you are aware of the implications of this.
When compressing data you get two numbers; the uncompressed and the compressed size.
Without compression anyone obtaining your data can only see one of these — the uncompressed size.
While t.
,
Compression Ratios
Some appliance vendors will promise, or even assume, a given compression ratio when calculating cost per TB.
With compression there is no guaranteed ratio above 1:1.
Different data types yield different compression ratios, and in our opinion there is no meaningful way to provide any average compression ratio.
Compression ratios should never be eval.
,
Configuring Compression in Minio
By default, on-disk compression is disabled in MinIO.
On-disk compression can be enabled or disabled at any time.
To enable on-disk compression use mc admin config set myminio compression enable=on.
This will enable compression for a preset number of extensions and MIME types.
By default these will include: You can inspect current settings with mc .
,
Does Minio compress a file?
Some data cannot be effectively compressed, such as:
- video or already compressed data
MinIO
does not compress common incompressible file types, even if they are specified in the compression configuration.
MinIO supports encrypting compressed objects but recommends against combining compression and encryption without a prior risk assessment.
,
Does Minio exclude gzip files?
By default MinIO forcefully excludes extensions of commonly incompressible data, such as:- gzip
audio video image files
It is possible to enable compression for all objects, except the excluded ones by setting the extension list and mime types to empty:. ,
How do I enable data compression in Minio subnet?
MinIO SUBNET users can log in and engage with our engineering and security teams to review encryption options.
To enable data compression, use mc admin config set to set the compression key enable option to on.
The following enables compression for new objects of the default types:
- Replace ALIAS with the alias of a configured MinIO deployment
,
Incompressible Objects
An important feature of modern compression is that it behaves well with pre-compressed data.
Traditionally pre-compressed data has been a problem for compression algorithms.
Often compressors would slow down unreasonably when confronted with incompressible data.
Many people therefore instinctively know that it is bad to recompress already compresse.
,
Introduction
Most data is stored in easy to read file formats for easy application interoperability.
While there are widely used formats for image, video, and sound compression, most other data is stored as text, as JSON, CSV, or other similar text-based formats.
Formats like Parquet, Avro, and ORC have optional compression, so typically these formats are alrea.
,
Seeking Compressed Files
A typical downside of compression is that the ability to skip within a file is lost.
The solution for this is to compress blocks independently and keep an index that maps a number of uncompressed offsets to compressed offsets where the decompression can start.
Compressing independent blocks will reduce compression slightly, but less with bigger blo.
,
Transparent Compression at Io Speed
Compression for MinIO has been developed to enable transparent compression without affecting the overall performance of the system. MinIO uses a compression method based on Snappy called S2.
It is compatible with Snappy content, but has two format extensions.
First of all, it allows bigger blocks than the 64KB blocks allowed for Snappy streams.
Thi.
,
Why should you use Minio?
MinIO offers the best-in-class compression scheme that allows for fully transparent compression of data on disk.
This can in many scenarios lead to a reduction in storage costs, simply by enabling compression.
GET and PUT performance should in all cases remain close to the same when compression is enabled.
