Supporting LZ4 as initramfs compressor

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Supporting LZ4 as initramfs compressor

Balint Reczey
Hi,

Initramfs-tools uses gzip compression by default which served us well
for quite some time but LZ4 offers way faster decompression while
making a only slightly bigger initramfs files.

On my old laptop the initramfs extraction time decreased from ~1.2s to ~0.24s:
(with lz4)
kernel: [    0.297726] Unpacking initramfs...
kernel: [    0.535061] Freeing initrd memory: 77940K
kernel: [    0.301637] Unpacking initramfs...
kernel: [    0.539109] Freeing initrd memory: 77940K
(with gzip)
kernel: [    0.273748] Unpacking initramfs...
kernel: [    1.490066] Freeing initrd memory: 57140K
kernel: [    0.281729] Unpacking initramfs...
kernel: [    1.498493] Freeing initrd memory: 57140K

The increase in the initrd.img size is ~14%:
(lz4)
-rw-r--r-- 1 root root 66709065 márc  19 14:24
/boot/initrd.img-4.15.0-12-generic
(gzip)
-rw-r--r-- 1 root root 58510993 márc  19 12:57
/boot/initrd.img-4.15.0-12-generic.bak

Initramfs creation speed also improved a bit from ~24s to ~21s wall clock time:
(lz4)
update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
14.97user 6.31system 0:20.47elapsed 103%CPU (0avgtext+0avgdata
22368maxresident)k
update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
15.18user 6.49system 0:20.48elapsed 105%CPU (0avgtext+0avgdata
22308maxresident)k
(gzip)
update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
18.23user 6.77system 0:23.61elapsed 105%CPU (0avgtext+0avgdata
22396maxresident)k
update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
18.38user 6.83system 0:23.82elapsed 105%CPU (0avgtext+0avgdata
22292maxresident)k

Base on the results I plan adding LZ4 compression support to
initramfs-tools as requested in LP: #1488620 [1] in the next days
without setting it as default and I propose setting LZ4 as default for
18.10.

I'm aware of the old problem of /boot filling up with kernels and
initramfs files but unattended-upgrades already removes old kernels
[2] and update-manager is planned to do the same starting with 18.04
[3] thus Ubuntu systems will have enough space in /boot to allow
slightly bigger initrd files.

Cheers,
Balint

[1] https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1488620
[2] https://launchpad.net/ubuntu/+source/unattended-upgrades/1.0ubuntu1
[3] https://code.launchpad.net/~rbalint/update-manager/remove-autoremovable-kernels/+merge/341599

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Julian Andres Klode
On Mon, Mar 19, 2018 at 02:59:24PM +0000, Balint Reczey wrote:

> Hi,
>
> Initramfs-tools uses gzip compression by default which served us well
> for quite some time but LZ4 offers way faster decompression while
> making a only slightly bigger initramfs files.
>
> On my old laptop the initramfs extraction time decreased from ~1.2s to ~0.24s:
> (with lz4)
> kernel: [    0.297726] Unpacking initramfs...
> kernel: [    0.535061] Freeing initrd memory: 77940K
> kernel: [    0.301637] Unpacking initramfs...
> kernel: [    0.539109] Freeing initrd memory: 77940K
> (with gzip)
> kernel: [    0.273748] Unpacking initramfs...
> kernel: [    1.490066] Freeing initrd memory: 57140K
> kernel: [    0.281729] Unpacking initramfs...
> kernel: [    1.498493] Freeing initrd memory: 57140K
>
> The increase in the initrd.img size is ~14%:
> (lz4)
> -rw-r--r-- 1 root root 66709065 márc  19 14:24
> /boot/initrd.img-4.15.0-12-generic
> (gzip)
> -rw-r--r-- 1 root root 58510993 márc  19 12:57
> /boot/initrd.img-4.15.0-12-generic.bak
>
> Initramfs creation speed also improved a bit from ~24s to ~21s wall clock time:
> (lz4)
> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> 14.97user 6.31system 0:20.47elapsed 103%CPU (0avgtext+0avgdata
> 22368maxresident)k
> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> 15.18user 6.49system 0:20.48elapsed 105%CPU (0avgtext+0avgdata
> 22308maxresident)k
> (gzip)
> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> 18.23user 6.77system 0:23.61elapsed 105%CPU (0avgtext+0avgdata
> 22396maxresident)k
> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> 18.38user 6.83system 0:23.82elapsed 105%CPU (0avgtext+0avgdata
> 22292maxresident)k

For me it was 16 -> 10 I think.

>
> Base on the results I plan adding LZ4 compression support to
> initramfs-tools as requested in LP: #1488620 [1] in the next days
> without setting it as default
>

+1

> and I propose setting LZ4 as default for 18.10.

We might have zstd support by that time (I hope), it might make sense
to use that then (better space/time tradeoff), but we'll have to see.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Balint Reczey
On Mon, Mar 19, 2018 at 3:12 PM, Julian Andres Klode
<[hidden email]> wrote:

> On Mon, Mar 19, 2018 at 02:59:24PM +0000, Balint Reczey wrote:
>> Hi,
>>
>> Initramfs-tools uses gzip compression by default which served us well
>> for quite some time but LZ4 offers way faster decompression while
>> making a only slightly bigger initramfs files.
>>
>> On my old laptop the initramfs extraction time decreased from ~1.2s to ~0.24s:
>> (with lz4)
>> kernel: [    0.297726] Unpacking initramfs...
>> kernel: [    0.535061] Freeing initrd memory: 77940K
>> kernel: [    0.301637] Unpacking initramfs...
>> kernel: [    0.539109] Freeing initrd memory: 77940K
>> (with gzip)
>> kernel: [    0.273748] Unpacking initramfs...
>> kernel: [    1.490066] Freeing initrd memory: 57140K
>> kernel: [    0.281729] Unpacking initramfs...
>> kernel: [    1.498493] Freeing initrd memory: 57140K
>>
>> The increase in the initrd.img size is ~14%:
>> (lz4)
>> -rw-r--r-- 1 root root 66709065 márc  19 14:24
>> /boot/initrd.img-4.15.0-12-generic
>> (gzip)
>> -rw-r--r-- 1 root root 58510993 márc  19 12:57
>> /boot/initrd.img-4.15.0-12-generic.bak
>>
>> Initramfs creation speed also improved a bit from ~24s to ~21s wall clock time:
>> (lz4)
>> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
>> 14.97user 6.31system 0:20.47elapsed 103%CPU (0avgtext+0avgdata
>> 22368maxresident)k
>> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
>> 15.18user 6.49system 0:20.48elapsed 105%CPU (0avgtext+0avgdata
>> 22308maxresident)k
>> (gzip)
>> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
>> 18.23user 6.77system 0:23.61elapsed 105%CPU (0avgtext+0avgdata
>> 22396maxresident)k
>> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
>> 18.38user 6.83system 0:23.82elapsed 105%CPU (0avgtext+0avgdata
>> 22292maxresident)k
>
> For me it was 16 -> 10 I think.
>
>>
>> Base on the results I plan adding LZ4 compression support to
>> initramfs-tools as requested in LP: #1488620 [1] in the next days
>> without setting it as default
>>
>
> +1
>
>> and I propose setting LZ4 as default for 18.10.
>
> We might have zstd support by that time (I hope), it might make sense
> to use that then (better space/time tradeoff), but we'll have to see.

That would also be an option, but I expect zstd to bring little speed
advantage over LZ4 here while LZ4 support could be easily backported
to releases with older kernels.

The proposed patch enabling lz4 uses lz4 -9. I tried lz4's default
compression, -1, but decompression speed difference was barely
noticeable - most likely due to copying being the bottleneck.

In fact I just realized that i copied the results created with lz4 -1,
where the compressed initrd size was 77940K.
The correct results for lz4 -9 are the following, taking the same
~0.24s to decompress:

kernel: [    0.285692] Unpacking initramfs...
kernel: [    0.518806] Freeing initrd memory: 65148K
kernel: [    0.289731] Unpacking initramfs...
kernel: [    0.522823] Freeing initrd memory: 65148K

Cheers,
Balint

--
Balint Reczey
Ubuntu & Debian Developer

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Marcos Alano
Last night I was trying to decrypt my laptop remotely. Work well
(after some research) and so on, but the point is I put Dropbear SSH
server inside initramfs and a faster decompress would be very
welcomed.

On Mon, Mar 19, 2018 at 2:19 PM, Balint Reczey
<[hidden email]> wrote:

> On Mon, Mar 19, 2018 at 3:12 PM, Julian Andres Klode
> <[hidden email]> wrote:
>> On Mon, Mar 19, 2018 at 02:59:24PM +0000, Balint Reczey wrote:
>>> Hi,
>>>
>>> Initramfs-tools uses gzip compression by default which served us well
>>> for quite some time but LZ4 offers way faster decompression while
>>> making a only slightly bigger initramfs files.
>>>
>>> On my old laptop the initramfs extraction time decreased from ~1.2s to ~0.24s:
>>> (with lz4)
>>> kernel: [    0.297726] Unpacking initramfs...
>>> kernel: [    0.535061] Freeing initrd memory: 77940K
>>> kernel: [    0.301637] Unpacking initramfs...
>>> kernel: [    0.539109] Freeing initrd memory: 77940K
>>> (with gzip)
>>> kernel: [    0.273748] Unpacking initramfs...
>>> kernel: [    1.490066] Freeing initrd memory: 57140K
>>> kernel: [    0.281729] Unpacking initramfs...
>>> kernel: [    1.498493] Freeing initrd memory: 57140K
>>>
>>> The increase in the initrd.img size is ~14%:
>>> (lz4)
>>> -rw-r--r-- 1 root root 66709065 márc  19 14:24
>>> /boot/initrd.img-4.15.0-12-generic
>>> (gzip)
>>> -rw-r--r-- 1 root root 58510993 márc  19 12:57
>>> /boot/initrd.img-4.15.0-12-generic.bak
>>>
>>> Initramfs creation speed also improved a bit from ~24s to ~21s wall clock time:
>>> (lz4)
>>> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
>>> 14.97user 6.31system 0:20.47elapsed 103%CPU (0avgtext+0avgdata
>>> 22368maxresident)k
>>> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
>>> 15.18user 6.49system 0:20.48elapsed 105%CPU (0avgtext+0avgdata
>>> 22308maxresident)k
>>> (gzip)
>>> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
>>> 18.23user 6.77system 0:23.61elapsed 105%CPU (0avgtext+0avgdata
>>> 22396maxresident)k
>>> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
>>> 18.38user 6.83system 0:23.82elapsed 105%CPU (0avgtext+0avgdata
>>> 22292maxresident)k
>>
>> For me it was 16 -> 10 I think.
>>
>>>
>>> Base on the results I plan adding LZ4 compression support to
>>> initramfs-tools as requested in LP: #1488620 [1] in the next days
>>> without setting it as default
>>>
>>
>> +1
>>
>>> and I propose setting LZ4 as default for 18.10.
>>
>> We might have zstd support by that time (I hope), it might make sense
>> to use that then (better space/time tradeoff), but we'll have to see.
>
> That would also be an option, but I expect zstd to bring little speed
> advantage over LZ4 here while LZ4 support could be easily backported
> to releases with older kernels.
>
> The proposed patch enabling lz4 uses lz4 -9. I tried lz4's default
> compression, -1, but decompression speed difference was barely
> noticeable - most likely due to copying being the bottleneck.
>
> In fact I just realized that i copied the results created with lz4 -1,
> where the compressed initrd size was 77940K.
> The correct results for lz4 -9 are the following, taking the same
> ~0.24s to decompress:
>
> kernel: [    0.285692] Unpacking initramfs...
> kernel: [    0.518806] Freeing initrd memory: 65148K
> kernel: [    0.289731] Unpacking initramfs...
> kernel: [    0.522823] Freeing initrd memory: 65148K
>
> Cheers,
> Balint
>
> --
> Balint Reczey
> Ubuntu & Debian Developer
>
> --
> ubuntu-devel mailing list
> [hidden email]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



--
Marcos H. Alano
Linux System Administrator
[hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Steve Langasek-6
In reply to this post by Balint Reczey
Hi Balint,

On Mon, Mar 19, 2018 at 02:59:24PM +0000, Balint Reczey wrote:

> Initramfs-tools uses gzip compression by default which served us well
> for quite some time but LZ4 offers way faster decompression while
> making a only slightly bigger initramfs files.

When people have previously discussed lz4 on IRC as a possible choice for
default compression algorithm, I had the impression that this was with the
expectation that the resulting initramfs files would be *smaller* than with
using gzip.

(I think.  It happens that names for compression algorithms are remarkably
unoriginal, so it's possible I've confused lz4 with another.)

But your data shows that lz4-compressed initramfs is definitely larger than
gzip, which means that there are tradeoffs here that should be fully
examined.

After all, an initramfs that's not compressed at all would take even less
time to decompress at boot (0s) but would occupy more space.  But you aren't
advocating for this, so there must be some reason you believe lz4 is more of
a sweet spot than gzip?


The first thing that I see missing from this analysis is the time it takes
for the firmware/bootloader to load the initramfs into memory.  I know from
experience that some bootloader filesystem drivers have quite poor
performance; and the time spent loading the initramfs into memory will scale
roughly linearly with the file size.  So any analysis of lz4 impact on boot
speed needs to take this into account.  We should show that the overall
bootspeed from bootloader to pid 1 has actually improved, and this should be
measured with multiple bootloader driver implementations (across
architectures; UEFI vs BIOS; possibly on multiple virtualization substrates
vs. x86 bare metal).


The second thing to consider is that, regardless of any improvements in our
autoremoval of kernels, we have had some historical default sizing for /boot
partitions in the installer which are now on the small side for even a fully
correct kernel upgrade path.

In trusty, the default (max) size for a /boot partition was 256M.  In
xenial, it was 512M.  In artful, we have bumped this up to 768M with a
minimum of 512M because of LP: #1716999.

The actual space consumed by the static files in the 4.13.0-7-generic kernel
in artful - not counting the current .efi.signed duplication which will
hopefully go away soon - is just under 13MiB.  My bootloader here is 8MiB,
but 10MiB is not out of the question in some configurations.  My initramfs
is 52MiB rather than the 55MiB in that bug, but again 55MiB is plausible -
and your own numbers seem to show 56MiB.

That means that anyone who installed with trusty, has /boot as a separate
partition, and has plymouth in the initramfs (such as for encrypted root
disk, which would be a common reason to have a separate /boot) has already
run out of space on their /boot while using gzip; so must already reinstall
or switch to a non-default initrd compression algorithm on upgrade.  This
can therefore be ignored for the choice between gzip and lz4 as default.

Anyone who installed with xenial is borderline today with artful;
56*4+3*13+10 == 273MiB, which is more than xenial's minimum /boot size of
256MiB but less than the max of 512MiB.  So some number of users have a boot
partition that's large enough to accommodate gzipped initrd, but will run
out of space once you switch to lz4 (64*4+3*13+10 == 305MiB).  And those
that don't run out of space immediately as a result of the switch to lz4
would still run out of space sooner as the kernel size grows (since the
kernel definitely seems to only grow over time with new drivers).

Systems installed with artful or newer seem to still be fine for a while,
with either gzip or lz4.

So there's a decision to be made about whether we care about upgrades
working with the default compression on systems installed with smaller boot,
and for how long.  If we decide this shouldn't block switching the default
compression, we also need to sort out how we will communicate this to
affected users - and in particular, how to avoid problems on upgrade when
the user runs out of disk space at the worst possible time.


> Base on the results I plan adding LZ4 compression support to
> initramfs-tools as requested in LP: #1488620 [1] in the next days
> without setting it as default and I propose setting LZ4 as default for
> 18.10.

Since this is a non-default option and doesn't introduce any new
dependencies, this seems fine.  It also doesn't seem urgent to me; in terms
of the upgrade path, it doesn't need to be supported in 18.04 in order to
consider making it the default in 18.10.

Cheers,
--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
[hidden email]                                     [hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Balint Reczey
Hi Steve,

On Wed, Mar 21, 2018 at 12:40 AM Steve Langasek
<[hidden email]> wrote:

>
> Hi Balint,
>
> On Mon, Mar 19, 2018 at 02:59:24PM +0000, Balint Reczey wrote:
>
> > Initramfs-tools uses gzip compression by default which served us well
> > for quite some time but LZ4 offers way faster decompression while
> > making a only slightly bigger initramfs files.
>
> When people have previously discussed lz4 on IRC as a possible choice for
> default compression algorithm, I had the impression that this was with the
> expectation that the resulting initramfs files would be *smaller* than with
> using gzip.
>
> (I think.  It happens that names for compression algorithms are remarkably
> unoriginal, so it's possible I've confused lz4 with another.)
>
> But your data shows that lz4-compressed initramfs is definitely larger than
> gzip, which means that there are tradeoffs here that should be fully
> examined.
>
> After all, an initramfs that's not compressed at all would take even less
> time to decompress at boot (0s) but would occupy more space.  But you aren't
> advocating for this, so there must be some reason you believe lz4 is more of
> a sweet spot than gzip?
>
>
> The first thing that I see missing from this analysis is the time it takes
> for the firmware/bootloader to load the initramfs into memory.  I know from
> experience that some bootloader filesystem drivers have quite poor
> performance; and the time spent loading the initramfs into memory will scale
> roughly linearly with the file size.  So any analysis of lz4 impact on boot
> speed needs to take this into account.  We should show that the overall
> bootspeed from bootloader to pid 1 has actually improved, and this should be
> measured with multiple bootloader driver implementations (across
> architectures; UEFI vs BIOS; possibly on multiple virtualization substrates
> vs. x86 bare metal).
>
>
> The second thing to consider is that, regardless of any improvements in our
> autoremoval of kernels, we have had some historical default sizing for /boot
> partitions in the installer which are now on the small side for even a fully
> correct kernel upgrade path.
>
> In trusty, the default (max) size for a /boot partition was 256M.  In
> xenial, it was 512M.  In artful, we have bumped this up to 768M with a
> minimum of 512M because of LP: #1716999.
>
> The actual space consumed by the static files in the 4.13.0-7-generic kernel
> in artful - not counting the current .efi.signed duplication which will
> hopefully go away soon - is just under 13MiB.  My bootloader here is 8MiB,
> but 10MiB is not out of the question in some configurations.  My initramfs
> is 52MiB rather than the 55MiB in that bug, but again 55MiB is plausible -
> and your own numbers seem to show 56MiB.
>
> That means that anyone who installed with trusty, has /boot as a separate
> partition, and has plymouth in the initramfs (such as for encrypted root
> disk, which would be a common reason to have a separate /boot) has already
> run out of space on their /boot while using gzip; so must already reinstall
> or switch to a non-default initrd compression algorithm on upgrade.  This
> can therefore be ignored for the choice between gzip and lz4 as default.
>
> Anyone who installed with xenial is borderline today with artful;
> 56*4+3*13+10 == 273MiB, which is more than xenial's minimum /boot size of
> 256MiB but less than the max of 512MiB.  So some number of users have a boot
> partition that's large enough to accommodate gzipped initrd, but will run
> out of space once you switch to lz4 (64*4+3*13+10 == 305MiB).  And those
> that don't run out of space immediately as a result of the switch to lz4
> would still run out of space sooner as the kernel size grows (since the
> kernel definitely seems to only grow over time with new drivers).
>
> Systems installed with artful or newer seem to still be fine for a while,
> with either gzip or lz4.
>
> So there's a decision to be made about whether we care about upgrades
> working with the default compression on systems installed with smaller boot,
> and for how long.  If we decide this shouldn't block switching the default
> compression, we also need to sort out how we will communicate this to
> affected users - and in particular, how to avoid problems on upgrade when
> the user runs out of disk space at the worst possible time.
>
>
> > Base on the results I plan adding LZ4 compression support to
> > initramfs-tools as requested in LP: #1488620 [1] in the next days
> > without setting it as default and I propose setting LZ4 as default for
> > 18.10.
>
> Since this is a non-default option and doesn't introduce any new
> dependencies, this seems fine.  It also doesn't seem urgent to me; in terms
> of the upgrade path, it doesn't need to be supported in 18.04 in order to
> consider making it the default in 18.10.

LZ4 compression is now available as a non-default option in Cosmic [2]
( \o/ :-) ).

Considering the potential problems with small boot partitions and even
more constrained embedded systems I'm not recommending making LZ4 as
the default, but making "auto" the default as discussed in [3] . Auto
would detect if there are space constraints and pick smallest (xz) or
fastest to boot (lz4) option.

Cheers,
Balint

[2] https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1488620/comments/8
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592519

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Colin Ian King-2
In reply to this post by Balint Reczey
Hi,

In a similar kind of exercise, I've been looking at the default
compression for the kernel and how this affects boot speed.

Data:
http://kernel.ubuntu.com/~cking/kernel-boot-speed-vs-compression.ods
(libreoffice spread sheet)

I've tested this on a fairly speedy desktop box and it is interesting to
see that lz4 is blisteringly fast at decompressing the kernel, next is
lzo followed by gzip.  xz, lzma and bzip2 are painfully slow in
decompression by comparison.

The trade off is compressed kernel size and hence how much time is
required to load a compressed kernel becomes important. Since gzip
compresses better than lz4 and lzo, one finds that a gzip compressed
kernel booted from a HDD boots slightly faster than lz4 and lzo. Where
as on a moderately fast SSD the faster load speed ends up with lz4 being
marginally faster than lzo and gzip.

So, the "best" kernel compression is hard to split between gzip/lzo/lz4
for a typical SSD based 3.4GHz box, and gzip is probably best for a HDD
based 3.4GHz box.  These results will differ on different H/W depending
on CPU speed, drive speed, bus speed and even how fragmented and/or
where the kernel is located on the boot device.

The next variable for me to look at is how CPU speed/architecture
changes boot performance, I hope to get around checking this out on a
Synquacer 24 proc ARM64 box in some time in the near future.

Colin


On 19/03/18 14:59, Balint Reczey wrote:

> Hi,
>
> Initramfs-tools uses gzip compression by default which served us well
> for quite some time but LZ4 offers way faster decompression while
> making a only slightly bigger initramfs files.
>
> On my old laptop the initramfs extraction time decreased from ~1.2s to ~0.24s:
> (with lz4)
> kernel: [    0.297726] Unpacking initramfs...
> kernel: [    0.535061] Freeing initrd memory: 77940K
> kernel: [    0.301637] Unpacking initramfs...
> kernel: [    0.539109] Freeing initrd memory: 77940K
> (with gzip)
> kernel: [    0.273748] Unpacking initramfs...
> kernel: [    1.490066] Freeing initrd memory: 57140K
> kernel: [    0.281729] Unpacking initramfs...
> kernel: [    1.498493] Freeing initrd memory: 57140K
>
> The increase in the initrd.img size is ~14%:
> (lz4)
> -rw-r--r-- 1 root root 66709065 márc  19 14:24
> /boot/initrd.img-4.15.0-12-generic
> (gzip)
> -rw-r--r-- 1 root root 58510993 márc  19 12:57
> /boot/initrd.img-4.15.0-12-generic.bak
>
> Initramfs creation speed also improved a bit from ~24s to ~21s wall clock time:
> (lz4)
> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> 14.97user 6.31system 0:20.47elapsed 103%CPU (0avgtext+0avgdata
> 22368maxresident)k
> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> 15.18user 6.49system 0:20.48elapsed 105%CPU (0avgtext+0avgdata
> 22308maxresident)k
> (gzip)
> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> 18.23user 6.77system 0:23.61elapsed 105%CPU (0avgtext+0avgdata
> 22396maxresident)k
> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> 18.38user 6.83system 0:23.82elapsed 105%CPU (0avgtext+0avgdata
> 22292maxresident)k
>
> Base on the results I plan adding LZ4 compression support to
> initramfs-tools as requested in LP: #1488620 [1] in the next days
> without setting it as default and I propose setting LZ4 as default for
> 18.10.
>
> I'm aware of the old problem of /boot filling up with kernels and
> initramfs files but unattended-upgrades already removes old kernels
> [2] and update-manager is planned to do the same starting with 18.04
> [3] thus Ubuntu systems will have enough space in /boot to allow
> slightly bigger initrd files.
>
> Cheers,
> Balint
>
> [1] https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1488620
> [2] https://launchpad.net/ubuntu/+source/unattended-upgrades/1.0ubuntu1
> [3] https://code.launchpad.net/~rbalint/update-manager/remove-autoremovable-kernels/+merge/341599
>


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Dimitri John Ledkov
Hey,

On Fri, 24 Aug 2018 at 14:11, Colin Ian King <[hidden email]> wrote:

>
> Hi,
>
> In a similar kind of exercise, I've been looking at the default
> compression for the kernel and how this affects boot speed.
>
> Data:
> http://kernel.ubuntu.com/~cking/kernel-boot-speed-vs-compression.ods
> (libreoffice spread sheet)
>
> I've tested this on a fairly speedy desktop box and it is interesting to
> see that lz4 is blisteringly fast at decompressing the kernel, next is
> lzo followed by gzip.  xz, lzma and bzip2 are painfully slow in
> decompression by comparison.
>
> The trade off is compressed kernel size and hence how much time is
> required to load a compressed kernel becomes important. Since gzip
> compresses better than lz4 and lzo, one finds that a gzip compressed
> kernel booted from a HDD boots slightly faster than lz4 and lzo. Where
> as on a moderately fast SSD the faster load speed ends up with lz4 being
> marginally faster than lzo and gzip.
>
> So, the "best" kernel compression is hard to split between gzip/lzo/lz4
> for a typical SSD based 3.4GHz box, and gzip is probably best for a HDD
> based 3.4GHz box.  These results will differ on different H/W depending
> on CPU speed, drive speed, bus speed and even how fragmented and/or
> where the kernel is located on the boot device.
>
> The next variable for me to look at is how CPU speed/architecture
> changes boot performance, I hope to get around checking this out on a
> Synquacer 24 proc ARM64 box in some time in the near future.
>

I see a lot of code in livecd-rootfs that tries hard to use lzma
compression for the initaller (first-boot) initrd.
On the classic, subsequent initrds get rebuild as gzip, and on core
lzma persists.
I do wonder what compression we should use by default for the
installer media, Ubuntu Core, cloud images.

Is the rationale for lzma installer initrds still valid? And what was
it? Smallest size initrd?


> Colin
>
>
> On 19/03/18 14:59, Balint Reczey wrote:
> > Hi,
> >
> > Initramfs-tools uses gzip compression by default which served us well
> > for quite some time but LZ4 offers way faster decompression while
> > making a only slightly bigger initramfs files.
> >
> > On my old laptop the initramfs extraction time decreased from ~1.2s to ~0.24s:
> > (with lz4)
> > kernel: [    0.297726] Unpacking initramfs...
> > kernel: [    0.535061] Freeing initrd memory: 77940K
> > kernel: [    0.301637] Unpacking initramfs...
> > kernel: [    0.539109] Freeing initrd memory: 77940K
> > (with gzip)
> > kernel: [    0.273748] Unpacking initramfs...
> > kernel: [    1.490066] Freeing initrd memory: 57140K
> > kernel: [    0.281729] Unpacking initramfs...
> > kernel: [    1.498493] Freeing initrd memory: 57140K
> >
> > The increase in the initrd.img size is ~14%:
> > (lz4)
> > -rw-r--r-- 1 root root 66709065 márc  19 14:24
> > /boot/initrd.img-4.15.0-12-generic
> > (gzip)
> > -rw-r--r-- 1 root root 58510993 márc  19 12:57
> > /boot/initrd.img-4.15.0-12-generic.bak
> >
> > Initramfs creation speed also improved a bit from ~24s to ~21s wall clock time:
> > (lz4)
> > update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> > 14.97user 6.31system 0:20.47elapsed 103%CPU (0avgtext+0avgdata
> > 22368maxresident)k
> > update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> > 15.18user 6.49system 0:20.48elapsed 105%CPU (0avgtext+0avgdata
> > 22308maxresident)k
> > (gzip)
> > update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> > 18.23user 6.77system 0:23.61elapsed 105%CPU (0avgtext+0avgdata
> > 22396maxresident)k
> > update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> > 18.38user 6.83system 0:23.82elapsed 105%CPU (0avgtext+0avgdata
> > 22292maxresident)k
> >
> > Base on the results I plan adding LZ4 compression support to
> > initramfs-tools as requested in LP: #1488620 [1] in the next days
> > without setting it as default and I propose setting LZ4 as default for
> > 18.10.
> >
> > I'm aware of the old problem of /boot filling up with kernels and
> > initramfs files but unattended-upgrades already removes old kernels
> > [2] and update-manager is planned to do the same starting with 18.04
> > [3] thus Ubuntu systems will have enough space in /boot to allow
> > slightly bigger initrd files.
> >
> > Cheers,
> > Balint
> >
> > [1] https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1488620
> > [2] https://launchpad.net/ubuntu/+source/unattended-upgrades/1.0ubuntu1
> > [3] https://code.launchpad.net/~rbalint/update-manager/remove-autoremovable-kernels/+merge/341599
> >
>
>
> --
> ubuntu-devel mailing list
> [hidden email]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



--
Regards,

Dimitri.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Dimitri John Ledkov
On Thu, 30 May 2019 at 11:35, Dimitri John Ledkov <[hidden email]> wrote:

>
> I see a lot of code in livecd-rootfs that tries hard to use lzma
> compression for the initaller (first-boot) initrd.
> On the classic, subsequent initrds get rebuild as gzip, and on core
> lzma persists.
> I do wonder what compression we should use by default for the
> installer media, Ubuntu Core, cloud images.
>
> Is the rationale for lzma installer initrds still valid? And what was
> it? Smallest size initrd?
>

Note subiquity dropped using lzma for the installer image to improve boot speed.
https://bugs.launchpad.net/subiquity/+bug/1750873

Which I find confusing from the spreadsheet graphs....
gzip is fast at kernel, but not initrd
lzma fast at initrd, but not kernel

As if lz4 kernel & xz initrd would yield the fastest boot time? That
sounds counter-intuitive. Unless for initrd, only the compressed size
matters and initrd decompression time does not matter?

Measurements until break=bottom is reached would be nice.

--
Regards,

Dimitri.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Paul Sladen-2
On Thu, 30 May 2019, Dimitri John Ledkov wrote:
> On Thu, 30 May 2019 at 11:35, Dimitri John Ledkov <[hidden email]> wrote:
> > On Fri, 24 Aug 2018 at 14:11, Colin Ian King <[hidden email]> wrote:
> > > http://kernel.ubuntu.com/~cking/kernel-boot-speed-vs-compression.ods
> > Is the rationale for lzma installer initrds still valid?
> gzip is fast at kernel, but not initrd
> lzma fast at initrd, but not kernel

Colin: would it be possible to re-run the statistics with a control
group (uncompressed kernel; uncompressed initramfs.cpio); this should
show the baseline I/O limits.

A second useful control, would be gzip/zlib in store-only mode
(0% compression): this would give a baseline for the overhead of
loading the data and running it via the decompression chain.

With those two baselines as a starting point; there are space (I/O,
storage) vs. time (load time, decode line) trade-offs---and
probably several local maxima.

> Measurements until break=bottom is reached would be nice.

Gzip/zlib is relatively well-understood stream protocol with literal
(raw data), LZ (lookback/string/matching), and Huffman (bit
squeezing).

Similiar to eg. MPEG, there is quite a bit of flexibility while
still being backwards compatible: multiple encoder and
decoders implementation, quite including hardware support.

It's only a hunch, but suspect that any speed goals can probably be
met with the existing Gzip/zlib; if not something is wrong(tm) or
sub-optimal somewhere.

(See the kernel decompression timing graph).

        -Paul




--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Benjamin Tegge
In reply to this post by Dimitri John Ledkov
Am Donnerstag, den 30.05.2019, 11:46 +0100 schrieb Dimitri John Ledkov:

> On Thu, 30 May 2019 at 11:35, Dimitri John Ledkov <[hidden email]>
> wrote:
> >
> > I see a lot of code in livecd-rootfs that tries hard to use lzma
> > compression for the initaller (first-boot) initrd.
> > On the classic, subsequent initrds get rebuild as gzip, and on core
> > lzma persists.
> > I do wonder what compression we should use by default for the
> > installer media, Ubuntu Core, cloud images.
> >
> > Is the rationale for lzma installer initrds still valid? And what
> > was
> > it? Smallest size initrd?
> >
>
> Note subiquity dropped using lzma for the installer image to improve
> boot speed.
> https://bugs.launchpad.net/subiquity/+bug/1750873
>
> Which I find confusing from the spreadsheet graphs....
> gzip is fast at kernel, but not initrd
> lzma fast at initrd, but not kernel
>
> As if lz4 kernel & xz initrd would yield the fastest boot time? That
> sounds counter-intuitive. Unless for initrd, only the compressed size
> matters and initrd decompression time does not matter?
>
> Measurements until break=bottom is reached would be nice.
>
> --
> Regards,
>
> Dimitri.
>

I initially suggested lz4 support a few years back in LP #1488620.  I
do not have insight in all the projects at Canonical/Ubuntu and what is
state of the art regarding compressors.  If you ask me today I would
recommend zstd, it is not intended to be as fast as lz4 and comparable
high speed decompressors, but faster than lzma and gzip.  I recently
found 7-Zip zstd[1] while looking for a solution that supports zstd on
Windows, may be the graphs in the project repository help you to find a
better solution and learn about recently developed compressors.

1: https://github.com/mcmilk/7-Zip-zstd


Best regards,
Benjamin


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Steve Langasek-6
In reply to this post by Dimitri John Ledkov
On Thu, May 30, 2019 at 11:35:26AM +0100, Dimitri John Ledkov wrote:
> I see a lot of code in livecd-rootfs that tries hard to use lzma
> compression for the initaller (first-boot) initrd.
> On the classic, subsequent initrds get rebuild as gzip, and on core
> lzma persists.
> I do wonder what compression we should use by default for the
> installer media, Ubuntu Core, cloud images.

> Is the rationale for lzma installer initrds still valid? And what was
> it? Smallest size initrd?

Yes, the rationale for using lzma on installer images was to prioritize size
over speed, because of fixed size limits for the installer image.

At the time this was implemented, we were still fitting the Ubuntu Desktop
installer onto a CD.

So this is definitely a decision that is subject to review.

> > On 19/03/18 14:59, Balint Reczey wrote:
> > > Hi,
> > >
> > > Initramfs-tools uses gzip compression by default which served us well
> > > for quite some time but LZ4 offers way faster decompression while
> > > making a only slightly bigger initramfs files.
> > >
> > > On my old laptop the initramfs extraction time decreased from ~1.2s to ~0.24s:
> > > (with lz4)
> > > kernel: [    0.297726] Unpacking initramfs...
> > > kernel: [    0.535061] Freeing initrd memory: 77940K
> > > kernel: [    0.301637] Unpacking initramfs...
> > > kernel: [    0.539109] Freeing initrd memory: 77940K
> > > (with gzip)
> > > kernel: [    0.273748] Unpacking initramfs...
> > > kernel: [    1.490066] Freeing initrd memory: 57140K
> > > kernel: [    0.281729] Unpacking initramfs...
> > > kernel: [    1.498493] Freeing initrd memory: 57140K
> > >
> > > The increase in the initrd.img size is ~14%:
> > > (lz4)
> > > -rw-r--r-- 1 root root 66709065 márc  19 14:24
> > > /boot/initrd.img-4.15.0-12-generic
> > > (gzip)
> > > -rw-r--r-- 1 root root 58510993 márc  19 12:57
> > > /boot/initrd.img-4.15.0-12-generic.bak
> > >
> > > Initramfs creation speed also improved a bit from ~24s to ~21s wall clock time:
> > > (lz4)
> > > update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> > > 14.97user 6.31system 0:20.47elapsed 103%CPU (0avgtext+0avgdata
> > > 22368maxresident)k
> > > update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> > > 15.18user 6.49system 0:20.48elapsed 105%CPU (0avgtext+0avgdata
> > > 22308maxresident)k
> > > (gzip)
> > > update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> > > 18.23user 6.77system 0:23.61elapsed 105%CPU (0avgtext+0avgdata
> > > 22396maxresident)k
> > > update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic
> > > 18.38user 6.83system 0:23.82elapsed 105%CPU (0avgtext+0avgdata
> > > 22292maxresident)k
> > >
> > > Base on the results I plan adding LZ4 compression support to
> > > initramfs-tools as requested in LP: #1488620 [1] in the next days
> > > without setting it as default and I propose setting LZ4 as default for
> > > 18.10.
> > >
> > > I'm aware of the old problem of /boot filling up with kernels and
> > > initramfs files but unattended-upgrades already removes old kernels
> > > [2] and update-manager is planned to do the same starting with 18.04
> > > [3] thus Ubuntu systems will have enough space in /boot to allow
> > > slightly bigger initrd files.
> > >
> > > Cheers,
> > > Balint
> > >
> > > [1] https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1488620
> > > [2] https://launchpad.net/ubuntu/+source/unattended-upgrades/1.0ubuntu1
> > > [3] https://code.launchpad.net/~rbalint/update-manager/remove-autoremovable-kernels/+merge/341599
> > >
> >
> >
> > --
> > ubuntu-devel mailing list
> > [hidden email]
> > Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
>
>
> --
> Regards,
>
> Dimitri.
>
> --
> ubuntu-devel mailing list
> [hidden email]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                   https://www.debian.org/
[hidden email]                                     [hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Seth Arnold
In reply to this post by Dimitri John Ledkov
On Thu, May 30, 2019 at 11:46:57AM +0100, Dimitri John Ledkov wrote:
> As if lz4 kernel & xz initrd would yield the fastest boot time? That

I'm lacking some context here, but I think building the initrds is already
too slow and I'm afraid xz on initrd rebuilds would be significantly
worse than lz4 or zstd or even low-compression levels of zlib.

Boot times are important but every now and then people do run apt upgrade
by hand and waiting forever to rebuild an initrd that may or may not be
used is pretty tedious. xz is great if the really slow compress times will
be made up for by better transfers or disk savings or similar.

Thanks

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Dimitri John Ledkov
On Fri, 31 May 2019 at 09:13, Seth Arnold <[hidden email]> wrote:

>
> On Thu, May 30, 2019 at 11:46:57AM +0100, Dimitri John Ledkov wrote:
> > As if lz4 kernel & xz initrd would yield the fastest boot time? That
>
> I'm lacking some context here, but I think building the initrds is already
> too slow and I'm afraid xz on initrd rebuilds would be significantly
> worse than lz4 or zstd or even low-compression levels of zlib.
>
> Boot times are important but every now and then people do run apt upgrade
> by hand and waiting forever to rebuild an initrd that may or may not be
> used is pretty tedious. xz is great if the really slow compress times will
> be made up for by better transfers or disk savings or similar.
>

I've tried to dig up past benchmarks in private discussions, public
discussions and kernel mailing list.

Zstd patches have not made it into the upstream kernel yet.

As used by mkinitramfs:
- lz4 is faster to compress than gzip
- lz4 is blazingly fast to decompress
- lzma is dog slow to compress and decompress, but is tiny
- lz4 size weight over gzip is marginal (14%) but imho worth the
improved boot time & initrd creation time
- xz is potentially even slower and even smaller than lzma

In places where size is an absolute premium (tiny embedded iot
devices) and performance is irrelevant, xz or lzma should be used.

In all other places, our performance profile is in favor of lz4.

Imho that includes the kernel image itself, thus we should consider switching:
- initramfs tools to default to lz4
- livecd-rootfs to default to lz4
- kernels to compress kernel image with lz4
- grub to include lz4 support

I shall proceed with changing the defaults on the above to improve our
responsiveness experience on installer, cloud, core and classic
devices.
If our firstboot & subsequent boot speed degrades or disk space
becomes a concern, we can look into tweaking these changes further.

--
Regards,

Dimitri.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Balint Reczey
Hi Dimitri,

Thanks for keeping this thread alive.

On Wed, Jun 5, 2019 at 2:16 PM Dimitri John Ledkov <[hidden email]> wrote:

>
> On Fri, 31 May 2019 at 09:13, Seth Arnold <[hidden email]> wrote:
> >
> > On Thu, May 30, 2019 at 11:46:57AM +0100, Dimitri John Ledkov wrote:
> > > As if lz4 kernel & xz initrd would yield the fastest boot time? That
> >
> > I'm lacking some context here, but I think building the initrds is already
> > too slow and I'm afraid xz on initrd rebuilds would be significantly
> > worse than lz4 or zstd or even low-compression levels of zlib.
> >
> > Boot times are important but every now and then people do run apt upgrade
> > by hand and waiting forever to rebuild an initrd that may or may not be
> > used is pretty tedious. xz is great if the really slow compress times will
> > be made up for by better transfers or disk savings or similar.
> >
>
> I've tried to dig up past benchmarks in private discussions, public
> discussions and kernel mailing list.
>
> Zstd patches have not made it into the upstream kernel yet.
>
> As used by mkinitramfs:
> - lz4 is faster to compress than gzip
> - lz4 is blazingly fast to decompress
> - lzma is dog slow to compress and decompress, but is tiny
> - lz4 size weight over gzip is marginal (14%) but imho worth the
> improved boot time & initrd creation time
> - xz is potentially even slower and even smaller than lzma
>
> In places where size is an absolute premium (tiny embedded iot
> devices) and performance is irrelevant, xz or lzma should be used.
>
> In all other places, our performance profile is in favor of lz4.
>
> Imho that includes the kernel image itself, thus we should consider switching:
> - initramfs tools to default to lz4
> - livecd-rootfs to default to lz4
> - kernels to compress kernel image with lz4
> - grub to include lz4 support

Please also note that it is not just the compressor that should be
tweaked but the parameters, too.
Initramfs-tools uses lz -9 because it provided the best space - speed
tradeoff, but comparisons testing the kernel compressors were using
the defaults which were not -9.
IMO lz4 (-9) would be the best choice for kernels, too, but to see
more accurate measurements it may be a good idea to run tests again
with setting compression level.

Cheers,
Balint

>
> I shall proceed with changing the defaults on the above to improve our
> responsiveness experience on installer, cloud, core and classic
> devices.
> If our firstboot & subsequent boot speed degrades or disk space
> becomes a concern, we can look into tweaking these changes further.
>
> --
> Regards,
>
> Dimitri.
>
> --
> ubuntu-devel mailing list
> [hidden email]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel


--
Balint Reczey
Ubuntu & Debian Developer

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Steve Langasek-6
In reply to this post by Dimitri John Ledkov
Hi Dimitri,

One point here:

On Wed, Jun 05, 2019 at 01:15:48PM +0100, Dimitri John Ledkov wrote:
> - lz4 size weight over gzip is marginal (14%) but imho worth the
> improved boot time & initrd creation time

A 14% increase in initramfs size is NOT marginal.  Since there are (and will
always be) various scenarios which require a separate /boot partition, we
have over several cycles been contending with how to ensure a clean upgrade
as kernel+initramfs sizes increase over time and push up against the space
limits of the boot partitions that have historically been created.

If you are making a change that increases the size of initramfses by 14%
across the board, you must also:

 - update the ubuntu-release-upgrader code to take this into account so
   users don't run out of space in the middle of the upgrade
   - possibly in a way that is smart enough to know if a user is already
     configuring initramfs-tools to use a non-default compressor, to avoid
     false-positives
 - check that the minimum size requirements for /boot that are encoded in
   the installers are still adequate, or if they need updating (and if they
   need updating, update this back to 18.04)
 - document this issue in the release notes

Thanks,
--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                   https://www.debian.org/
[hidden email]                                     [hidden email]

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Supporting LZ4 as initramfs compressor

Dimitri John Ledkov
On Wed, 5 Jun 2019 at 19:59, Steve Langasek <[hidden email]> wrote:

>
> Hi Dimitri,
>
> One point here:
>
> On Wed, Jun 05, 2019 at 01:15:48PM +0100, Dimitri John Ledkov wrote:
> > - lz4 size weight over gzip is marginal (14%) but imho worth the
> > improved boot time & initrd creation time
>
> A 14% increase in initramfs size is NOT marginal.  Since there are (and will
> always be) various scenarios which require a separate /boot partition, we
> have over several cycles been contending with how to ensure a clean upgrade
> as kernel+initramfs sizes increase over time and push up against the space
> limits of the boot partitions that have historically been created.
>
> If you are making a change that increases the size of initramfses by 14%
> across the board, you must also:
>
>  - update the ubuntu-release-upgrader code to take this into account so
>    users don't run out of space in the middle of the upgrade
>    - possibly in a way that is smart enough to know if a user is already
>      configuring initramfs-tools to use a non-default compressor, to avoid
>      false-positives
>  - check that the minimum size requirements for /boot that are encoded in
>    the installers are still adequate, or if they need updating (and if they
>    need updating, update this back to 18.04)
>  - document this issue in the release notes

agree.

--
Regards,

Dimitri.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel