kybaaz/linux-xiaomi-chiron
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Changes since last update:

Browse source 
- support multiple devices for multi-layer container images;
 
  - support the secondary compression head;
 
  - support readmore decompression strategy;
 
  - support new LZMA algorithm (specifically called MicroLZMA);
 
  - some bugfixes & cleanups.
 -----BEGIN PGP SIGNATURE-----
 
 iIcEABYIAC8WIQThPAmQN9sSA0DVxtI5NzHcH7XmBAUCYX8j7hEceGlhbmdAa2Vy
 bmVsLm9yZwAKCRA5NzHcH7XmBE+SAQChAmAUav03OQujm8PvVNX7VUGusGNvww8E
 qu5+zasC8wEArypW2Z75ZZ3IZNPCk6QWFlaC2I5Xnz7NNl0OGPKOCAg=
 =DZQ4
 -----END PGP SIGNATURE-----

Merge tag 'erofs-for-5.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs

Pull erofs updates from Gao Xiang:
 "There are some new features available for this cycle. Firstly, EROFS
  LZMA algorithm support, specifically called MicroLZMA, is available as
  an option for embedded devices, LiveCDs and/or as the secondary
  auxiliary compression algorithm besides the primary algorithm in one
  file.

  In order to better support the LZMA fixed-sized output compression,
  especially for 4KiB pcluster size (which has lowest memory pressure
  thus useful for memory-sensitive scenarios), Lasse introduced a new
  LZMA header/container format called MicroLZMA to minimize the original
  LZMA1 header (for example, we don't need to waste 4-byte dictionary
  size and another 8-byte uncompressed size, which can be calculated by
  fs directly, for each pcluster) and enable EROFS fixed-sized output
  compression.

  Note that MicroLZMA can also be later used by other things in addition
  to EROFS too where wasting minimal amount of space for headers is
  important and it can be only compiled by enabling XZ_DEC_MICROLZMA.
  MicroLZMA has been supported by the latest upstream XZ embedded [1] &
  XZ utils [2], apply the latest related XZ embedded upstream patches by
  the XZ author Lasse here.

  Secondly, multiple device is also supported in this cycle, which is
  designed for multi-layer container images. By working together with
  inter-layer data deduplication and compression, we can achieve the
  next high-performance container image solution. Our team will announce
  the new Nydus container image service [3] implementation with new RAFS
  v6 (EROFS-compatible) format in Open Source Summit 2021 China [4]
  soon.

  Besides, the secondary compression head support and readmore
  decompression strategy are also included in this cycle. There are also
  some minor bugfixes and cleanups, as always.

  Summary:

   - support multiple devices for multi-layer container images;

   - support the secondary compression head;

   - support readmore decompression strategy;

   - support new LZMA algorithm (specifically called MicroLZMA);

   - some bugfixes & cleanups"

* tag 'erofs-for-5.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs:
  erofs: don't trigger WARN() when decompression fails
  erofs: get rid of ->lru usage
  erofs: lzma compression support
  erofs: rename some generic methods in decompressor
  lib/xz, lib/decompress_unxz.c: Fix spelling in comments
  lib/xz: Add MicroLZMA decoder
  lib/xz: Move s->lzma.len = 0 initialization to lzma_reset()
  lib/xz: Validate the value before assigning it to an enum variable
  lib/xz: Avoid overlapping memcpy() with invalid input with in-place decompression
  erofs: introduce readmore decompression strategy
  erofs: introduce the secondary compression head
  erofs: get compression algorithms directly on mapping
  erofs: add multiple device support
  erofs: decouple basic mount options from fs_context
  erofs: remove the fast path of per-CPU buffer decompression
This commit is contained in:
Linus Torvalds 2021-11-01 11:39:22 -07:00
commit 67a135b80e
25 changed files with 1282 additions and 321 deletions
Download patch file Download diff file Expand all files Collapse all files

106
include/linux/xz.h
View file

@ -233,6 +233,112 @@ XZ_EXTERN void xz_dec_reset(struct xz_dec *s);
*/
XZ_EXTERN void xz_dec_end(struct xz_dec *s);
/*
* Decompressor for MicroLZMA, an LZMA variant with a very minimal header.
* See xz_dec_microlzma_alloc() below for details.
*
* These functions aren't used or available in preboot code and thus aren't
* marked with XZ_EXTERN. This avoids warnings about static functions that
* are never defined.
*/
/**
* struct xz_dec_microlzma - Opaque type to hold the MicroLZMA decoder state
*/
struct xz_dec_microlzma;
/**
* xz_dec_microlzma_alloc() - Allocate memory for the MicroLZMA decoder
* @mode XZ_SINGLE or XZ_PREALLOC
* @dict_size LZMA dictionary size. This must be at least 4 KiB and
* at most 3 GiB.
*
* In contrast to xz_dec_init(), this function only allocates the memory
* and remembers the dictionary size. xz_dec_microlzma_reset() must be used
* before calling xz_dec_microlzma_run().
*
* The amount of allocated memory is a little less than 30 KiB with XZ_SINGLE.
* With XZ_PREALLOC also a dictionary buffer of dict_size bytes is allocated.
*
* On success, xz_dec_microlzma_alloc() returns a pointer to
* struct xz_dec_microlzma. If memory allocation fails or
* dict_size is invalid, NULL is returned.
*
* The compressed format supported by this decoder is a raw LZMA stream
* whose first byte (always 0x00) has been replaced with bitwise-negation
* of the LZMA properties (lc/lp/pb) byte. For example, if lc/lp/pb is
* 3/0/2, the first byte is 0xA2. This way the first byte can never be 0x00.
* Just like with LZMA2, lc + lp <= 4 must be true. The LZMA end-of-stream
* marker must not be used. The unused values are reserved for future use.
* This MicroLZMA header format was created for use in EROFS but may be used
* by others too.
*/
extern struct xz_dec_microlzma *xz_dec_microlzma_alloc(enum xz_mode mode,
uint32_t dict_size);
/**
* xz_dec_microlzma_reset() - Reset the MicroLZMA decoder state
* @s Decoder state allocated using xz_dec_microlzma_alloc()
* @comp_size Compressed size of the input stream
* @uncomp_size Uncompressed size of the input stream. A value smaller
* than the real uncompressed size of the input stream can
* be specified if uncomp_size_is_exact is set to false.
* uncomp_size can never be set to a value larger than the
* expected real uncompressed size because it would eventually
* result in XZ_DATA_ERROR.
* @uncomp_size_is_exact This is an int instead of bool to avoid
* requiring stdbool.h. This should normally be set to true.
* When this is set to false, error detection is weaker.
*/
extern void xz_dec_microlzma_reset(struct xz_dec_microlzma *s,
uint32_t comp_size, uint32_t uncomp_size,
int uncomp_size_is_exact);
/**
* xz_dec_microlzma_run() - Run the MicroLZMA decoder
* @s Decoder state initialized using xz_dec_microlzma_reset()
* @b: Input and output buffers
*
* This works similarly to xz_dec_run() with a few important differences.
* Only the differences are documented here.
*
* The only possible return values are XZ_OK, XZ_STREAM_END, and
* XZ_DATA_ERROR. This function cannot return XZ_BUF_ERROR: if no progress
* is possible due to lack of input data or output space, this function will
* keep returning XZ_OK. Thus, the calling code must be written so that it
* will eventually provide input and output space matching (or exceeding)
* comp_size and uncomp_size arguments given to xz_dec_microlzma_reset().
* If the caller cannot do this (for example, if the input file is truncated
* or otherwise corrupt), the caller must detect this error by itself to
* avoid an infinite loop.
*
* If the compressed data seems to be corrupt, XZ_DATA_ERROR is returned.
* This can happen also when incorrect dictionary, uncompressed, or
* compressed sizes have been specified.
*
* With XZ_PREALLOC only: As an extra feature, b->out may be NULL to skip over
* uncompressed data. This way the caller doesn't need to provide a temporary
* output buffer for the bytes that will be ignored.
*
* With XZ_SINGLE only: In contrast to xz_dec_run(), the return value XZ_OK
* is also possible and thus XZ_SINGLE is actually a limited multi-call mode.
* After XZ_OK the bytes decoded so far may be read from the output buffer.
* It is possible to continue decoding but the variables b->out and b->out_pos
* MUST NOT be changed by the caller. Increasing the value of b->out_size is
* allowed to make more output space available; one doesn't need to provide
* space for the whole uncompressed data on the first call. The input buffer
* may be changed normally like with XZ_PREALLOC. This way input data can be
* provided from non-contiguous memory.
*/
extern enum xz_ret xz_dec_microlzma_run(struct xz_dec_microlzma *s,
struct xz_buf *b);
/**
* xz_dec_microlzma_end() - Free the memory allocated for the decoder state
* @s: Decoder state allocated using xz_dec_microlzma_alloc().
* If s is NULL, this function does nothing.
*/
extern void xz_dec_microlzma_end(struct xz_dec_microlzma *s);
/*
* Standalone build (userspace build or in-kernel build for boot time use)
* needs a CRC32 implementation. For normal in-kernel use, kernel's own