README - pub/scm/linux/kernel/git/xiang/erofs-utils - Git at Google


            
            erofs-utils

            
            ===========


            Userspace tools for EROFS filesystem, currently including:


              mkfs.erofs    filesystem formatter

            
              erofsfuse     FUSE daemon alternative

            
              dump.erofs    filesystem analyzer

            
              fsck.erofs    filesystem compatibility & consistency checker as well

            
                            as extractor


            EROFS filesystem overview

            
            -------------------------


            EROFS filesystem stands for Enhanced Read-Only File System.  It aims to

            
            form a generic read-only filesystem solution for various read-only use

            
            cases instead of just focusing on storage space saving without

            
            considering any side effects of runtime performance.


            Typically EROFS could be considered in the following use scenarios:

            
              - Firmwares in performance-sensitive systems, such as system

            
                partitions of Android smartphones;


              - Mountable immutable images such as container images for effective

            
                metadata & data access compared with tar, cpio or other local

            
                filesystems (e.g. ext4, XFS, btrfs, etc.)


              - FSDAX-enabled rootfs for secure containers (Linux 5.15+);


              - Live CDs which need a set of files with another high-performance

            
                algorithm to optimize startup time; others files for archival

            
                purposes only are not needed;


              - and more.


            Note that all EROFS metadata is uncompressed by design, so that you

            
            could take EROFS as a drop-in read-only replacement of ext4, XFS,

            
            btrfs, etc. without any compression-based dependencies and EROFS can

            
            bring more effective filesystem accesses to users with reduced

            
            metadata.


            For more details of EROFS filesystem itself, please refer to:

            
            https://www.kernel.org/doc/html/next/filesystems/erofs.html


            For more details on how to build erofs-utils, see `docs/INSTALL.md`.


            For more details about filesystem performance, see

            
            `docs/PERFORMANCE.md`.


            mkfs.erofs

            
            ----------


            Two main kinds of EROFS images can be generated: (un)compressed images.


             - For uncompressed images, there will be no compressed files in these

            
               images.  However, an EROFS image can contain files which consist of

            
               various aligned data blocks and then a tail that is stored inline in

            
               order to compact images [1].


             - For compressed images, it will try to use the given algorithms first

            
               for each regular file and see if storage space can be saved with

            
               compression. If not, it will fall back to an uncompressed file.


            Note that EROFS supports per-file compression configuration, proper

            
            configuration options need to be enabled to parse compressed files by

            
            the Linux kernel.


            How to generate EROFS images

            
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~


            Compression algorithms could be specified with the command-line option

            
            `-z` to build a compressed EROFS image from a local directory:

            
             $ mkfs.erofs -zlz4hc foo.erofs.img foo/


            Supported algorithms by the Linux kernel:

            
             - LZ4 (Linux 5.3+);

            
             - LZMA (Linux 5.16+);

            
             - DEFLATE (Linux 6.6+);

            
             - Zstandard (Linux 6.10+).


            Alternatively, generate an uncompressed EROFS from a local directory:

            
             $ mkfs.erofs foo.erofs.img foo/


            Additionally, you can specify a higher compression level to get a

            
            (slightly) smaller image than the default level:

            
             $ mkfs.erofs -zlz4hc,12 foo.erofs.img foo/


            Multi-threaded support can be explicitly enabled with the ./configure

            
            option `--enable-multithreading`; otherwise, single-threaded compression

            
            will be used for now.  It may take more time on multiprocessor platforms

            
            if multi-threaded support is not enabled.


            Currently, both `-Efragments` (not `-Eall-fragments`) and `-Ededupe`

            
            don't support multi-threading due to time limitations.


            Reproducible builds

            
            ~~~~~~~~~~~~~~~~~~~


            Reproducible builds are typically used for verification and security,

            
            ensuring the same binaries/distributions to be reproduced in a

            
            deterministic way.


            Images generated by the same version of `mkfs.erofs` will be identical

            
            to previous runs if the same input is specified, and the same options

            
            are used.


            Specifically, variable timestamps and filesystem UUIDs can result in

            
            unreproducible EROFS images.  `-T` and `-U` can be used to fix them.


            How to generate EROFS big pcluster images (Linux 5.13+)

            
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


            By default, EROFS formatter compresses data into separate one-block

            
            (e.g. 4KiB) filesystem physical clusters for outstanding random read

            
            performance.  In other words, each EROFS filesystem block can be

            
            independently decompressed.  However, other similar filesystems

            
            typically compress data into "blocks" of 128KiB or more for much smaller

            
            images.  Users may prefer smaller images for archiving purposes, even if

            
            random performance is compromised with those configurations, and even

            
            worse when using 4KiB blocks.


            In order to fulfill users' needs, big plusters has been introduced

            
            since Linux 5.13, in which each physical clusters will be more than one

            
            blocks.


            Specifically, `-C` is used to specify the maximum size of each pcluster

            
            in bytes:

            
             $ mkfs.erofs -zlz4hc -C65536 foo.erofs.img foo/


            Thus, in this case, pcluster sizes can be up to 64KiB.


            Note that large pcluster size can degrade random performance (though it

            
            may improve sequential read performance for typical storage devices), so

            
            please evaluate carefully in advance.  Alternatively, you can make

            
            per-(sub)file compression strategies according to file access patterns

            
            if needed.


            How to generate EROFS images with multiple algorithms

            
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


            It's possible to generate an EROFS image with files in different

            
            algorithms due to various purposes.  For example, LZMA for archival

            
            purposes and LZ4 for runtime purposes.


            In order to use alternative algorithms, just specify two or more

            
            compressing configurations together separated by ':' like below:

            
                -zlzma:lz4hc,12:lzma,9 -C32768


            Although mkfs still choose the first one by default, you could try to

            
            write a compress-hints file like below:

            
                4096  1 .*\.so$

            
                32768 2 .*\.txt$

            
                4096    sbin/.*$

            
                16384 0 .*


            and specify with `--compress-hints=` so that ".so" files will use

            
            "lz4hc,12" compression with 4k pclusters, ".txt" files will use

            
            "lzma,9" compression with 32k pclusters, files  under "/sbin" will use

            
            the default "lzma" compression with 4k plusters and other files will

            
            use "lzma" compression with 16k pclusters.


            Note that the largest pcluster size should be specified with the "-C"

            
            option (here 32k pcluster size), otherwise all larger pclusters will be

            
            limited.


            How to generate well-compressed EROFS images

            
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


            Even if EROFS is not designed for such purposes in the beginning, it

            
            could still produce some smaller images (not always) compared to other

            
            approaches with better performance (see `docs/PERFORMANCE.md`).  In

            
            order to build well-compressed EROFS images, try the following options:

            
             -C1048576                     (5.13+)

            
             -Eztailpacking                (5.16+)

            
             -Efragments / -Eall-fragments ( 6.1+);

            
             -Ededupe                      ( 6.1+).


            Also EROFS uses lz4hc level 9 by default, whereas some other approaches

            
            use lz4hc level 12 by default.  So please explicitly specify

            
            `-zlz4hc,12 ` for comparison purposes.


            How to generate legacy EROFS images (Linux 4.19+)

            
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


            Decompression inplace and compacted indexes have been introduced in

            
            Linux v5.3, which are not forward-compatible with older kernels.


            In order to generate _legacy_ EROFS images for old kernels,

            
            consider adding "-E legacy-compress" to the command line, e.g.


             $ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/


            For Linux kernel >= 5.3, legacy EROFS images are _NOT recommended_

            
            due to runtime performance loss compared with non-legacy images.


            Obsoleted erofs.mkfs

            
            ~~~~~~~~~~~~~~~~~~~~


            There is an original erofs.mkfs version developed by Li Guifu,

            
            which was replaced by the new erofs-utils implementation.


            git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs


            PLEASE NOTE: This version is highly _NOT recommended_ now.


            erofsfuse

            
            ---------


            erofsfuse is introduced to support EROFS format for various platforms

            
            (including older linux kernels) and new on-disk features iteration.

            
            It can also be used as an unpacking tool for unprivileged users.


            It supports fixed-sized output decompression *without* any in-place

            
            I/O or in-place decompression optimization. Also like the other FUSE

            
            implementations, it suffers from most common performance issues (e.g.

            
            significant I/O overhead, double caching, etc.)


            Therefore, NEVER use it if performance is the top concern.


            How to mount an EROFS image with erofsfuse

            
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


            As the other FUSE implementations, it's quite easy to mount by using

            
            erofsfuse, e.g.:

            
             $ erofsfuse foo.erofs.img foo/


            Alternatively, to make it run in foreground (with debugging level 3):

            
             $ erofsfuse -f --dbglevel=3 foo.erofs.img foo/


            To debug erofsfuse (also automatically run in foreground):

            
             $ erofsfuse -d foo.erofs.img foo/


            To unmount an erofsfuse mountpoint as a non-root user:

            
             $ fusermount -u foo/


            dump.erofs and fsck.erofs

            
            -------------------------


            dump.erofs and fsck.erofs are used to analyze, check, and extract

            
            EROFS filesystems. Note that extended attributes and ACLs are still

            
            unsupported when extracting images with fsck.erofs.


            Note that fragment extraction with fsck.erofs could be slow now and

            
            it needs to be optimized later.  If you are interested, contribution

            
            is, as always, welcome.


            Contribution

            
            ------------


            erofs-utils is a part of EROFS filesystem project, which is completely

            
            community-driven open source software.  If you have interest in EROFS,

            
            feel free to send feedback and/or patches to:

            
              linux-erofs mailing list   <linux-erofs@lists.ozlabs.org>


            Comments

            
            --------


            [1] According to the EROFS on-disk format, the tail blocks of files

            
                could be inlined aggressively with their metadata (called

            
                tail-packing) in order to minimize the extra I/Os and the storage

            
                space.
	erofs-utils
	===========

	Userspace tools for EROFS filesystem, currently including:

	mkfs.erofs filesystem formatter
	erofsfuse FUSE daemon alternative
	dump.erofs filesystem analyzer
	fsck.erofs filesystem compatibility & consistency checker as well
	as extractor


	EROFS filesystem overview
	-------------------------

	EROFS filesystem stands for Enhanced Read-Only File System. It aims to
	form a generic read-only filesystem solution for various read-only use
	cases instead of just focusing on storage space saving without
	considering any side effects of runtime performance.

	Typically EROFS could be considered in the following use scenarios:
	- Firmwares in performance-sensitive systems, such as system
	partitions of Android smartphones;

	- Mountable immutable images such as container images for effective
	metadata & data access compared with tar, cpio or other local
	filesystems (e.g. ext4, XFS, btrfs, etc.)

	- FSDAX-enabled rootfs for secure containers (Linux 5.15+);

	- Live CDs which need a set of files with another high-performance
	algorithm to optimize startup time; others files for archival
	purposes only are not needed;

	- and more.

	Note that all EROFS metadata is uncompressed by design, so that you
	could take EROFS as a drop-in read-only replacement of ext4, XFS,
	btrfs, etc. without any compression-based dependencies and EROFS can
	bring more effective filesystem accesses to users with reduced
	metadata.

	For more details of EROFS filesystem itself, please refer to:
	https://www.kernel.org/doc/html/next/filesystems/erofs.html

	For more details on how to build erofs-utils, see `docs/INSTALL.md`.

	For more details about filesystem performance, see
	`docs/PERFORMANCE.md`.


	mkfs.erofs
	----------

	Two main kinds of EROFS images can be generated: (un)compressed images.

	- For uncompressed images, there will be no compressed files in these
	images. However, an EROFS image can contain files which consist of
	various aligned data blocks and then a tail that is stored inline in
	order to compact images [1].

	- For compressed images, it will try to use the given algorithms first
	for each regular file and see if storage space can be saved with
	compression. If not, it will fall back to an uncompressed file.

	Note that EROFS supports per-file compression configuration, proper
	configuration options need to be enabled to parse compressed files by
	the Linux kernel.

	How to generate EROFS images
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	Compression algorithms could be specified with the command-line option
	`-z` to build a compressed EROFS image from a local directory:
	$ mkfs.erofs -zlz4hc foo.erofs.img foo/

	Supported algorithms by the Linux kernel:
	- LZ4 (Linux 5.3+);
	- LZMA (Linux 5.16+);
	- DEFLATE (Linux 6.6+);
	- Zstandard (Linux 6.10+).

	Alternatively, generate an uncompressed EROFS from a local directory:
	$ mkfs.erofs foo.erofs.img foo/

	Additionally, you can specify a higher compression level to get a
	(slightly) smaller image than the default level:
	$ mkfs.erofs -zlz4hc,12 foo.erofs.img foo/

	Multi-threaded support can be explicitly enabled with the ./configure
	option `--enable-multithreading`; otherwise, single-threaded compression
	will be used for now. It may take more time on multiprocessor platforms
	if multi-threaded support is not enabled.

	Currently, both `-Efragments` (not `-Eall-fragments`) and `-Ededupe`
	don't support multi-threading due to time limitations.

	Reproducible builds
	~~~~~~~~~~~~~~~~~~~

	Reproducible builds are typically used for verification and security,
	ensuring the same binaries/distributions to be reproduced in a
	deterministic way.

	Images generated by the same version of `mkfs.erofs` will be identical
	to previous runs if the same input is specified, and the same options
	are used.

	Specifically, variable timestamps and filesystem UUIDs can result in
	unreproducible EROFS images. `-T` and `-U` can be used to fix them.

	How to generate EROFS big pcluster images (Linux 5.13+)
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	By default, EROFS formatter compresses data into separate one-block
	(e.g. 4KiB) filesystem physical clusters for outstanding random read
	performance. In other words, each EROFS filesystem block can be
	independently decompressed. However, other similar filesystems
	typically compress data into "blocks" of 128KiB or more for much smaller
	images. Users may prefer smaller images for archiving purposes, even if
	random performance is compromised with those configurations, and even
	worse when using 4KiB blocks.

	In order to fulfill users' needs, big plusters has been introduced
	since Linux 5.13, in which each physical clusters will be more than one
	blocks.

	Specifically, `-C` is used to specify the maximum size of each pcluster
	in bytes:
	$ mkfs.erofs -zlz4hc -C65536 foo.erofs.img foo/

	Thus, in this case, pcluster sizes can be up to 64KiB.

	Note that large pcluster size can degrade random performance (though it
	may improve sequential read performance for typical storage devices), so
	please evaluate carefully in advance. Alternatively, you can make
	per-(sub)file compression strategies according to file access patterns
	if needed.

	How to generate EROFS images with multiple algorithms
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	It's possible to generate an EROFS image with files in different
	algorithms due to various purposes. For example, LZMA for archival
	purposes and LZ4 for runtime purposes.

	In order to use alternative algorithms, just specify two or more
	compressing configurations together separated by ':' like below:
	-zlzma:lz4hc,12:lzma,9 -C32768

	Although mkfs still choose the first one by default, you could try to
	write a compress-hints file like below:
	4096 1 .*\.so$
	32768 2 .*\.txt$
	4096 sbin/.*$
	16384 0 .*

	and specify with `--compress-hints=` so that ".so" files will use
	"lz4hc,12" compression with 4k pclusters, ".txt" files will use
	"lzma,9" compression with 32k pclusters, files under "/sbin" will use
	the default "lzma" compression with 4k plusters and other files will
	use "lzma" compression with 16k pclusters.

	Note that the largest pcluster size should be specified with the "-C"
	option (here 32k pcluster size), otherwise all larger pclusters will be
	limited.

	How to generate well-compressed EROFS images
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	Even if EROFS is not designed for such purposes in the beginning, it
	could still produce some smaller images (not always) compared to other
	approaches with better performance (see `docs/PERFORMANCE.md`). In
	order to build well-compressed EROFS images, try the following options:
	-C1048576 (5.13+)
	-Eztailpacking (5.16+)
	-Efragments / -Eall-fragments ( 6.1+);
	-Ededupe ( 6.1+).

	Also EROFS uses lz4hc level 9 by default, whereas some other approaches
	use lz4hc level 12 by default. So please explicitly specify
	`-zlz4hc,12 ` for comparison purposes.

	How to generate legacy EROFS images (Linux 4.19+)
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	Decompression inplace and compacted indexes have been introduced in
	Linux v5.3, which are not forward-compatible with older kernels.

	In order to generate _legacy_ EROFS images for old kernels,
	consider adding "-E legacy-compress" to the command line, e.g.

	$ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/

	For Linux kernel >= 5.3, legacy EROFS images are _NOT recommended_
	due to runtime performance loss compared with non-legacy images.

	Obsoleted erofs.mkfs
	~~~~~~~~~~~~~~~~~~~~

	There is an original erofs.mkfs version developed by Li Guifu,
	which was replaced by the new erofs-utils implementation.

	git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs

	PLEASE NOTE: This version is highly _NOT recommended_ now.


	erofsfuse
	---------

	erofsfuse is introduced to support EROFS format for various platforms
	(including older linux kernels) and new on-disk features iteration.
	It can also be used as an unpacking tool for unprivileged users.

	It supports fixed-sized output decompression without any in-place
	I/O or in-place decompression optimization. Also like the other FUSE
	implementations, it suffers from most common performance issues (e.g.
	significant I/O overhead, double caching, etc.)

	Therefore, NEVER use it if performance is the top concern.

	How to mount an EROFS image with erofsfuse
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	As the other FUSE implementations, it's quite easy to mount by using
	erofsfuse, e.g.:
	$ erofsfuse foo.erofs.img foo/

	Alternatively, to make it run in foreground (with debugging level 3):
	$ erofsfuse -f --dbglevel=3 foo.erofs.img foo/

	To debug erofsfuse (also automatically run in foreground):
	$ erofsfuse -d foo.erofs.img foo/

	To unmount an erofsfuse mountpoint as a non-root user:
	$ fusermount -u foo/


	dump.erofs and fsck.erofs
	-------------------------

	dump.erofs and fsck.erofs are used to analyze, check, and extract
	EROFS filesystems. Note that extended attributes and ACLs are still
	unsupported when extracting images with fsck.erofs.

	Note that fragment extraction with fsck.erofs could be slow now and
	it needs to be optimized later. If you are interested, contribution
	is, as always, welcome.


	Contribution
	------------

	erofs-utils is a part of EROFS filesystem project, which is completely
	community-driven open source software. If you have interest in EROFS,
	feel free to send feedback and/or patches to:
	linux-erofs mailing list <linux-erofs@lists.ozlabs.org>


	Comments
	--------

	[1] According to the EROFS on-disk format, the tail blocks of files
	could be inlined aggressively with their metadata (called
	tail-packing) in order to minimize the extra I/Os and the storage
	space.