ZPool not mounting at boot

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

ZPool not mounting at boot

Liam Proven
I did a lot of flailing around on this one, and I _may_ have solved
it, in a slapshot way. Not sure yet.

I run Ubuntu Server 20.04 64-bit on a Raspberry Pi 4, as a home NAS
server. It has 4 x 2TB USB3 drives in a ZFS RAID on a USB 3 hub which
also powers the RasPi. No fans, passive heatsink/case, silent except
for disk accesses.

I built this in May, described here:
https://liam-on-linux.livejournal.com/71093.html

It's been notable that Focal on the RasPi 4 was not very stable at first.
• My ethernet connection works but traffic pauses for from seconds to minutes
• wifi worked better, after I wrote a netplan YAML, but would randomly
drop the IPv4 address
• I had a lot of CPU activity from multipathd
• I had a bunch of snaps mounted continuing GNOME stuff - on a
headless server with no GUI
• no ZFS binaries were included or in the repo -- it all had to be
built from source from scratch. First time took 45 min with all 4
cores pegged

I removed snapd, multipathd, some bit of the GNOME search backend I
forget. This got me a hundred meg or so of RAM back, ended dozens of
always-active threads, and took idle CPU usage down to about 4%.

It ran usefully, but I had to connect to the wifi IP for consistent
performance. Ditto for management with ssh. Over wifi, it can take
writes of about 1GB/min which is quite good.

I use Webmin to monitor it and netatalk so my Mac can back up to it
with Time Machine.

Over the months, gradually, with updates, things have improved.

Ethernet performance seems stable now: no freezes.
Wifi keeps its IPv4 address for weeks at a time.
Automatic unattended-updates run regularly and email me the results.

But recently I noticed that I could connect to my shares after a
reboot. On investigation, the zpool is not mounting. I can do it
manually with zpool import vol0 but I should not need this.

Last night, I did an update, then `shutdown -r now`. It did not come
back up. No IP leases, nothing.

I had to pull the plug. That's a first and not a good one.

I did some Googling and found this: a StackExchange answer that lists
a load of systemd services that should be autostarting:
https://unix.stackexchange.com/questions/338260/zfs-mounting-only-one-of-my-pools-on-boot

I did them all. Only 1 gave an error. Last reboot, it did mount my
RAID successfully, but this reboot took about 10min and was rather
nerve-wracking -- in that time, I could not ssh into the box.

Now it's running happily again though, and my Mac is preparing a Time
Machine backup for the first time in 36 hours.


--
Liam Proven – Profile: https://about.me/liamproven
Email: [hidden email] – gMail/gTalk/gHangouts: [hidden email]
Twitter/Facebook/LinkedIn/Flickr: lproven – Skype: liamproven
UK: +44 7939-087884 – ČR (+ WhatsApp/Telegram/Signal): +420 702 829 053

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: ZPool not mounting at boot

ubuntu-users mailing list
On Sat, 13 Feb 2021 01:45:59 +0100, Liam Proven wrote:
>On investigation, the zpool is not mounting. I can do it manually with
>zpool import vol0 but I should not need this.

"Warning: Do not run zpool import pool! This will import your pools
using /dev/sd? which will lead to problems the next time you rearrange
your drives. This may be as simple as rebooting with a USB drive left
in the machine." -
https://wiki.archlinux.org/index.php/ZFS#Importing_a_pool_created_by_id

>Last reboot, it did mount my RAID successfully, but this reboot took
>about 10min

"Your boot time can be significantly impacted if you update your
intitramfs (eg when doing a kernel update) when you have additional but
non-permanently attached pools imported because these pools will get
added to your initramfs zpool.cache and ZFS will attempt to import
these extra pools on every boot, regardless of whether you have
exported it and removed it from your regular zpool.cache." -
https://wiki.archlinux.org/index.php/ZFS#Fix_slow_boot_caused_by_failed_import_of_unavailable_pools_in_the_initramfs_zpool.cache

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: ZPool not mounting at boot

Chris Green
In reply to this post by Liam Proven
On Sat, Feb 13, 2021 at 01:45:59AM +0100, Liam Proven wrote:

[snip tale of Ubuntu 20.04 on Pi]

I set up a Pi 4 with an external 8Tb drive as a NAS last September.  I
installed the minimum 'Raspberry Pi OS Lite' and it has run
faultlessly ever since, not a single problem (except those of my own
doing).

The Raspberry Pi OS is similar enough to Ubuntu (especially when
console/headless) for it not to be worth installing a specifically
Ubuntu OS IMHO.

--
Chris Green

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: ZPool not mounting at boot

Liam Proven
In reply to this post by ubuntu-users mailing list
On Sat, 13 Feb 2021 at 09:38, Ralf Mardorf via ubuntu-users
<[hidden email]> wrote:

>
> On Sat, 13 Feb 2021 01:45:59 +0100, Liam Proven wrote:
> >On investigation, the zpool is not mounting. I can do it manually with
> >zpool import vol0 but I should not need this.
>
> "Warning: Do not run zpool import pool! This will import your pools
> using /dev/sd? which will lead to problems the next time you rearrange
> your drives. This may be as simple as rebooting with a USB drive left
> in the machine." -
> https://wiki.archlinux.org/index.php/ZFS#Importing_a_pool_created_by_id

Interesting. I think I did originally build it using /dev/sd? entries;
I can't remember now.

> "Your boot time can be significantly impacted if you update your
> intitramfs (eg when doing a kernel update)

This takes ages anyway, as it must recompile the ZFS modules for the new kernel.

>  when you have additional but
> non-permanently attached pools

I have no additional pools at all.

But worth knowing -- thank you.

--
Liam Proven – Profile: https://about.me/liamproven
Email: [hidden email] – gMail/gTalk/gHangouts: [hidden email]
Twitter/Facebook/LinkedIn/Flickr: lproven – Skype: liamproven
UK: +44 7939-087884 – ČR (+ WhatsApp/Telegram/Signal): +420 702 829 053

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: ZPool not mounting at boot

Liam Proven
In reply to this post by Chris Green
On Sat, 13 Feb 2021 at 11:59, Chris Green <[hidden email]> wrote:
>
> I set up a Pi 4 with an external 8Tb drive as a NAS last September.  I
> installed the minimum 'Raspberry Pi OS Lite' and it has run
> faultlessly ever since, not a single problem (except those of my own
> doing).

But that is a single drive. I have an array.

> The Raspberry Pi OS is similar enough to Ubuntu (especially when
> console/headless) for it not to be worth installing a specifically
> Ubuntu OS IMHO.

I do not run RasPi OS on any of my RasPis and never have.

AFAIK, RaspBian does not support ZFS, and nor does Debian upstream.

I have used Linux `mdraid` a fair bit and it is somewhat
labour-intensive. I wanted to give ZFS a try and I have been
impressed: it is fast and easy. The tool creates the array, manages
partitioning too, formats it, makes a mountpoint and mounts it for
you, all automatically, in a single step.

I don't personally need snapshots or copy-on-write or any of that, but
just as for an array-management tool it is more convenient than
mdraid.


--
Liam Proven – Profile: https://about.me/liamproven
Email: [hidden email] – gMail/gTalk/gHangouts: [hidden email]
Twitter/Facebook/LinkedIn/Flickr: lproven – Skype: liamproven
UK: +44 7939-087884 – ČR (+ WhatsApp/Telegram/Signal): +420 702 829 053

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: ZPool not mounting at boot

Chris Green
On Sat, Feb 13, 2021 at 02:19:00PM +0100, Liam Proven wrote:
> On Sat, 13 Feb 2021 at 11:59, Chris Green <[hidden email]> wrote:
> >
> > I set up a Pi 4 with an external 8Tb drive as a NAS last September.  I
> > installed the minimum 'Raspberry Pi OS Lite' and it has run
> > faultlessly ever since, not a single problem (except those of my own
> > doing).
>
> But that is a single drive. I have an array.
>
Yes, one of the reasons I steer away from RAID, you have to go to
extreme lengths with it to actually gain security IMHO.  As it's a
*backup* system I only need what it contains if one of my other
systems fails.  The chances of simultaneous failures of both main and
backup systems is small enough for me not to want to go to further
complexity on the backup which, anyway, I believe makes it more rather
than less likely to go wrong.

> > The Raspberry Pi OS is similar enough to Ubuntu (especially when
> > console/headless) for it not to be worth installing a specifically
> > Ubuntu OS IMHO.
>
> I do not run RasPi OS on any of my RasPis and never have.
>
I've swapped back and forth between that and Ubuntu and find the
differences so slight, particularly when running headless, that it
makes very little difference.

> AFAIK, RaspBian does not support ZFS, and nor does Debian upstream.
>
I can see little difference between what's available on my xubuntu
systems and what's available on the Pi systems, e.g. on the Pi
'apt-cache search zfs' shows loads of stuff, including:-

    libzfs4linux - OpenZFS filesystem library for Linux
    libzfs4linux-dbgsym - debug symbols for libzfs4linux
    libzfsbootenv1linux - OpenZFS filesystem library for Linux
    libzfsbootenv1linux-dbgsym - debug symbols for libzfsbootenv1linux
    libzfslinux-dev - OpenZFS filesystem development files for Linux
    libzpool4linux - OpenZFS pool library for Linux
    ...
    ...
    zfs-dkms - OpenZFS filesystem kernel modules for Linux
    zfs-dracut - OpenZFS root filesystem capabilities for Linux - dracut
    zfs-initramfs - OpenZFS root filesystem capabilities for Linux - initramfs


--
Chris Green

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: ZPool not mounting at boot

Liam Proven
On Sat, 13 Feb 2021 at 14:52, Chris Green <[hidden email]> wrote:

> Yes, one of the reasons I steer away from RAID, you have to go to
> extreme lengths with it to actually gain security IMHO.  As it's a
> *backup* system I only need what it contains if one of my other
> systems fails.  The chances of simultaneous failures of both main and
> backup systems is small enough for me not to want to go to further
> complexity on the backup which, anyway, I believe makes it more rather
> than less likely to go wrong.

So long as the MTBF is longer than one hard disk, I favour it. I've
been using it for decades now.  My old home server was a cast-off HP
ProLiant ML110 (1st gen, 32-bit P4) with a Dell PERC UltraIDE RAID
controller in it, and 6 x 80GB IDE disks. Biggest I could amass a pile
of them and some spares.

That replaced an old Gateway with 4 x 40GB.

Then I replaced the HP with a MicroServer with 4 x 300GB drives. Now,
the RasPi with 4 x 2TB drives. They cost me about £150 from Morgan
Computers.


> I've swapped back and forth between that and Ubuntu and find the
> differences so slight, particularly when running headless, that it
> makes very little difference.

Well, TBH, I put RaspBian x86 on my Sony Vaio P a fortnight ago. It's
as lightweight as CrunchBang++ -- uses 200MB running. It's quite quick
even on this fairly feeble dual-core Atom, maxed out with 2GB RAM. I
haven't played much but it went on easily and it works.

There aren't many 32-bit distros left now. :-(

> I can see little difference between what's available on my xubuntu
> systems and what's available on the Pi systems, e.g. on the Pi
> 'apt-cache search zfs' shows loads of stuff, including:-

Huh. Well, I did say "AFAIK".

I know it's not in SUSE or openSUSE because of licensing concerns, and
for the same reasons I suspect it's not in any RH distro.

--
Liam Proven – Profile: https://about.me/liamproven
Email: [hidden email] – gMail/gTalk/gHangouts: [hidden email]
Twitter/Facebook/LinkedIn/Flickr: lproven – Skype: liamproven
UK: +44 7939-087884 – ČR (+ WhatsApp/Telegram/Signal): +420 702 829 053

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: ZPool not mounting at boot

J. Paul Bissonnette-2
Excuse me for interrupting do you have any experience with freNAS? I was thinking of using it.
--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: ZPool not mounting at boot

Liam Proven
On Sun, 14 Feb 2021 at 01:28, J. Paul Bissonnette <[hidden email]> wrote:
>
> Excuse me for interrupting do you have any experience with freNAS? I was thinking of using it.

Personally, very little. However, friends use it and recommend it.

Note:
* it's been discontinued -- now it's called TrueNAS, and TrueNAS Core
is the free version.
* It's based on FreeBSD, not Linux

There is a FreeBSD for the RasPi, but building a NAS server is a
strictly DIY operation and I don't know FreeBSD well. So I chose
Ubuntu which I am very familiar with.

I am only aware of one NAS OS for the Raspberry Pi: Open Media Vault.

I have looked at OMV before and it wasn't really what I wanted. E.g. I
don't stream media from the Pi at all. It's based on Debian, and it
doesn't use ZFS.

Also, at the time I built my server, OMV did not support the RasPi 4 yet.

If at some point there is a real native TrueNAS port for the Pi,
perhaps for the 8GB Pi 4, then I might redo this server on that basis.
It would be a lot less work.

However, mine works, and I'm happy with it. It's silent, it was cheap
to build, reliability and performance are passable -- OK for home use.

--
Liam Proven – Profile: https://about.me/liamproven
Email: [hidden email] – gMail/gTalk/gHangouts: [hidden email]
Twitter/Facebook/LinkedIn/Flickr: lproven – Skype: liamproven
UK: +44 7939-087884 – ČR (+ WhatsApp/Telegram/Signal): +420 702 829 053

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: ZPool not mounting at boot

J. Paul Bissonnette-2
On Sun, 14 Feb 2021 01:59:31 +0100
Liam Proven <[hidden email]> wrote:

> However, mine works, and I'm happy with it. It's silent, it was cheap
> to build, reliability and performance are passable -- OK for home use.
>

Have you published any where how to build one?
A Synology DS1813+ I have dies and their "Support team" politely
informed me that after the warranty period there is no parts, no
service and absolutely no support. Now I have an almost mint paper
weight sans Mobo.

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: ZPool not mounting at boot

Liam Proven
On Sun, 14 Feb 2021 at 14:58, J. Paul Bissonnette <[hidden email]> wrote:
> >
>
> Have you published any where how to build one?

It was in my first message, the root of the thread:

> I built this in May, described here:
> https://liam-on-linux.livejournal.com/71093.html

> A Synology DS1813+ I have dies and their "Support team" politely
> informed me that after the warranty period there is no parts, no
> service and absolutely no support. Now I have an almost mint paper
> weight sans Mobo.

To be honest, yes, this is why I do not have or use dedicated NAS
devices. They are quite expensive, and many run proprietary OSes, so
if the motherboard dies, you can't get the data off the disks.

I have certainly used proprietary hardware RAID controllers in
production servers, but _because_ they are production machines, they
were regularly backed up and most lived on UPSes.

My home boxes do not. I am too cheap. :-)

I have a vague plan to upgrade the RAM in my elderly HP Microserver,
put bigger disks in it, and run TrueNAS Core on it. I was considering
running VMware ESXi on it and having it do other things too, but wiser
people than I told me that this is violating the KISS principle and
that if I want to do things like that, TrueNAS is not what I want.

In the past I have often found cheap clever technical solutions to
avoid buying fancy dedicated hardware. Some worked, some did not.

One of the distinguishing factors is that a clever solution that does
one thing well, using cheap off-the-shelf bits and FOSS, can often
succeed. If you look at the commercial/proprietary products, often
they are based on some off-the-shelf tech – such as Linux running on a
cheap CPU.

If so, sure, why not do it yourself? :-)

--
Liam Proven – Profile: https://about.me/liamproven
Email: [hidden email] – gMail/gTalk/gHangouts: [hidden email]
Twitter/Facebook/LinkedIn/Flickr: lproven – Skype: liamproven
UK: +44 7939-087884 – ČR (+ WhatsApp/Telegram/Signal): +420 702 829 053

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users