LVM: How to access a foreign volume group

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

LVM: How to access a foreign volume group

Volker Wysk
Hi!

I'm about to set up an LVM cache (for my encrypted root file system). This involves attaching a cache pool LV to the origin LV to be cached, as the last step:

lvconvert --type cache --cachepool kubuntu-vg/cachedatalv kubuntu-vg/root

I've got it working in a virtual machine, but I'm hesitant to apply the procedure to my production system. It would be bad, if I had to restore it from a backup, because I have installed and configured a lot of things.

So, in case something goes wrong, I want to be able to start a rescue system and undo the attaching of the cache pool LV to the origin LV. This would be the command to do this:

lvconvert --uncache kubuntu-vg/root

But the question is: How to access the LVM setup of the other (production) system from the rescue system? The rescue system is encrypted too, but I could make it unencrypted, non-LVM, if needed.

How to instruct LVM to operate on another installation? When I start the rescue system, which I've installed on an USB stick, I can access only the native LVM installation, like it should be.

Also, both volume groups have the same name. Because I installed both from the Kubuntu 16.04 install ISO (from an USB stick), and this assigns "kubuntu-vg" to the VG name. (I'm using "Guided - use entire disk and setu up encrypted LVM" in the "Disk Setup" step. "Manual" doesn't work, and looks broken).

Thanks,
Volker


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: LVM: How to access a foreign volume group

Volker Wysk
Hi

I've found the recovery mode in the Ubuntu Users wiki. I'll read some more now.

Bye
V.W.


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|

Re: LVM: How to access a foreign volume group

Xen
In reply to this post by Volker Wysk
Volker Wysk schreef op 04-12-2017 2:03:

> I'm about to set up an LVM cache (for my encrypted root file system).

I suggest the following in /etc/initramfs-tools/hooks/dmcache:

-----------
#!/bin/sh

if [ "$1" = "prereqs" ]; then
     exit 0
fi

. /usr/share/initramfs-tools/hook-functions

# pdata_tools is the cache_check executable
copy_exec /usr/sbin/pdata_tools

# you have to use relative symlinks here:
for f in /usr/sbin/cache*; do ln -sr ${DESTDIR}/usr/sbin/pdata_tools
${DESTDIR}$f; done

# safest is just to copy all of the device mapper modules:
copy_modules_dir kernel/drivers/md
-----------

But I assume you have already done this?

> It would be bad, if I had to restore it from a backup, because I have
> installed and configured a lot of things.

The greatest risk is that the logical volume for root is not activated
because of a missing module or executable in your initrd.

> How to instruct LVM to operate on another installation? When I start
> the rescue system, which I've installed on an USB stick, I can access
> only the native LVM installation, like it should be.

That's because your main system is encrypted.

Otherwise LVM would start destroying your system (maybe).

> Also, both volume groups have the same name.

Yes you won't be able to activate both volume groups very well until you
rename the one in the rescue system.

This isn't very hard.

BEFORE you do any cryptsetup open on the main system, run:

vgrename kubuntu-vg rescue-vg

Then you must verify that /etc/fstab in the RESCUE system contains no
references to "kubuntu-vg" and you might also have to rerun
"update-grub" if you want to keep it this way.

That is all you need to do to rename the volume group.

After that you can open the crypt and it won't conflict.

> Because I installed both
> from the Kubuntu 16.04 install ISO (from an USB stick), and this
> assigns "kubuntu-vg" to the VG name.

You are wise to say so, the LVM of 16.04 is not very good with
conflicts.

(16.10 is better).


Make sure /etc/initramfs-tools/hooks/dmcache is executable:

chmod +x /etc/initramfs-tools/hooks/dmcache

Also rerun update-initramfs -u.

After, verify that the initramfs contains everything you need:

lsinitramfs /boot/initrd* | grep "pdata\|cache"

It has to contain at least:

usr/sbin/cache_restore
usr/sbin/cache_repair
usr/sbin/cache_metadata_size
usr/sbin/cache_dump
usr/sbin/cache_check
usr/sbin/pdata_tools
lib/modules/4.10.0-40-generic/kernel/drivers/md/dm-cache-cleaner.ko
lib/modules/4.10.0-40-generic/kernel/drivers/md/dm-cache-smq.ko
lib/modules/4.10.0-40-generic/kernel/drivers/md/dm-cache.ko

Or similar.

I assume you will be running this from your installed system. So you
have only one chance to get it right before you need to reboot into the
live or rescue environment.

If you end up on an initrd prompt (busybox) because your system doesn't
boot, try to run:

   vgchange -ay
   vgchange -ay

   exit

To continue booting.

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|

Re: LVM: How to access a foreign volume group

Xen
In reply to this post by Volker Wysk
Volker Wysk schreef op 04-12-2017 2:03:
> Hi!
>
> I'm about to set up an LVM cache (for my encrypted root file system).

Also I assume you will also encrypt your cache volumes?

This is not easy.

I don't think the initrd by default opens non-root encrypted thingies.

So you would need these steps:

1) create second container on cache disk (SSD)
2) open it
3) pvcreate
4) vgextend
5) create your cache pool

6) add a key to the container
7) add the container to /etc/crypttab including its key
8) create another hook in /etc/initramfs-tools/hooks for copying the key
into the initrd
9) create a script in /etc/initramfs-tools/scripts/local-top that will
explicitly open the container because I don't think it happens
otherwise.

10) cause this script to also activate the cache volumes (maybe)

11) create a copy of /usr/share/initramfs-tools/scripts/local-top/lvm2
in /etc/initramfs-tools/scripts/local-top

12) edit this to have your new "cache_unlock" script as prereq

13) rerun update-initramfstools -u

You really need all that to have an encrypted cache.

Here are some of the steps.


1) cryptsetup luksFormat /dev/sdb1   (assuming sdb is your SSD)
2) cryptsetup open /dev/sdb1 cache_crypt
3) pvcreate /dev/mapper/cache_crypt
4) vgextend kubuntu-vg /dev/mapper/cache_crypt
5) lvcreate kubuntu-vg --cache-pool bla bla bla  
/dev/mapper/cache_crypt

6) dd if=/dev/random of=/root/cache.key bs=1M count=1
6) cryptsetup luksAddKey /dev/sdb1 /root/cache.key

7) echo "cache_crypt /dev/disk/by-uuid/$(blkid /dev/sdb1 -s UUID o
value) /cache.key luks,keyscript=/bin/cat" >> /etc/crypttab

8) cat > /etc/initramfs-tools/hooks/cachekey << EOF
#!/bin/sh

if [ "$1" = "prereqs" ]; then
     exit 0
fi

cp /root/cache.key $DESTDIR
EOF

8) chmod +x /etc/initramfs-tools/hooks/cachekey
8) update-initramfs -u

The rest would have to wait.

This is the only way to (automatically) open a 2nd container containing
the encrypted cache volumes for your cached root.

Buuh.

It doesn't become easier does it.

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: LVM: How to access a foreign volume group

Volker Wysk
Am Samstag, 9. Dezember 2017, 22:08:54 CET schrieb Xen:

> Volker Wysk schreef op 04-12-2017 2:03:
> > Hi!
> >
> > I'm about to set up an LVM cache (for my encrypted root file system).
>
> Also I assume you will also encrypt your cache volumes?
>
> This is not easy.
>
> I don't think the initrd by default opens non-root encrypted thingies.
>
> So you would need these steps:
>
> (... ... ...)

I'm afraid, your help goes to waste here, since I've already got it running, in a virtual machine. It wasn't easy, but I've managed to set it up. Your steps look familiar.

The reason I'm worrying how to restore the system from backup, is because something might go wrong, when I apply the procedure to my production system, and I won't be able to boot it any longer. I want to know how to recover _before_ I break my machine. I've also looked at GRUB's recovey mode.

Happy hacking,
Volker


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|

Re: LVM: How to access a foreign volume group

Xen
Volker Wysk schreef op 10-12-2017 2:38:

> Am Samstag, 9. Dezember 2017, 22:08:54 CET schrieb Xen:
>> Volker Wysk schreef op 04-12-2017 2:03:
>> > Hi!
>> >
>> > I'm about to set up an LVM cache (for my encrypted root file system).
>>
>> Also I assume you will also encrypt your cache volumes?
>>
>> This is not easy.
>>
>> I don't think the initrd by default opens non-root encrypted thingies.
>>
>> So you would need these steps:
>>
>> (... ... ...)
>
> I'm afraid, your help goes to waste here, since I've already got it
> running, in a virtual machine. It wasn't easy, but I've managed to set
> it up. Your steps look familiar.
>
> The reason I'm worrying how to restore the system from backup, is
> because something might go wrong, when I apply the procedure to my
> production system, and I won't be able to boot it any longer. I want
> to know how to recover _before_ I break my machine. I've also looked
> at GRUB's recovey mode.

Oh well then I apologize, it just seemed to me a person capable of
setting this up would also know how to use a Live DVD ;-).

In a live DVD there is nothing more to do than install
thin-provisioning-tools and running cryptsetup open. That's it.

That's why I eh... misjudged you in that sense. Regards.

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: LVM: How to access a foreign volume group

Volker Wysk
In reply to this post by Xen
Hi

I've used these instructions (in German):
https://www.thomas-krenn.com/de/wiki/LVM_Caching_mit_SSDs_einrichten

I've worked it out, although there were some issues, which needed fixing. It works now, in a virtual machine.

What you've described in you mail, below, looks familiar to me:


Am Samstag, 9. Dezember 2017, 21:32:40 CET schrieb Xen:

> Volker Wysk schreef op 04-12-2017 2:03:
>
> > I'm about to set up an LVM cache (for my encrypted root file system).
>
> I suggest the following in /etc/initramfs-tools/hooks/dmcache:
>
> -----------
> #!/bin/sh
>
> if [ "$1" = "prereqs" ]; then
>      exit 0
> fi
>
> . /usr/share/initramfs-tools/hook-functions
>
> # pdata_tools is the cache_check executable
> copy_exec /usr/sbin/pdata_tools
>
> # you have to use relative symlinks here:
> for f in /usr/sbin/cache*; do ln -sr ${DESTDIR}/usr/sbin/pdata_tools
> ${DESTDIR}$f; done
>
> # safest is just to copy all of the device mapper modules:
> copy_modules_dir kernel/drivers/md
> -----------
>
> But I assume you have already done this?

The procedure, which I've followed, involves something similar. See above.

 
> > It would be bad, if I had to restore it from a backup, because I have
> > installed and configured a lot of things.
>
> The greatest risk is that the logical volume for root is not activated
> because of a missing module or executable in your initrd.

Precisely. That's what I've encountered.


> > How to instruct LVM to operate on another installation? When I start
> > the rescue system, which I've installed on an USB stick, I can access
> > only the native LVM installation, like it should be.
>
> That's because your main system is encrypted.
>
> Otherwise LVM would start destroying your system (maybe).
>
> > Also, both volume groups have the same name.
>
> Yes you won't be able to activate both volume groups very well until you
> rename the one in the rescue system.
>
> This isn't very hard.
>
> BEFORE you do any cryptsetup open on the main system, run:
>
> vgrename kubuntu-vg rescue-vg
>
> Then you must verify that /etc/fstab in the RESCUE system contains no
> references to "kubuntu-vg" and you might also have to rerun
> "update-grub" if you want to keep it this way.
>
> That is all you need to do to rename the volume group.
>
> After that you can open the crypt and it won't conflict.

Yes, that's what I was looking for. However, for now I'll use a non-encrypted, non-LVM rescue system.

 

> > Because I installed both
> > from the Kubuntu 16.04 install ISO (from an USB stick), and this
> > assigns "kubuntu-vg" to the VG name.
>
> You are wise to say so, the LVM of 16.04 is not very good with
> conflicts.
>
> (16.10 is better).
>
>
> Make sure /etc/initramfs-tools/hooks/dmcache is executable:
>
> chmod +x /etc/initramfs-tools/hooks/dmcache
>
> Also rerun update-initramfs -u.
>
> After, verify that the initramfs contains everything you need:
>
> lsinitramfs /boot/initrd* | grep "pdata\|cache"
>
> It has to contain at least:
>
> usr/sbin/cache_restore
> usr/sbin/cache_repair
> usr/sbin/cache_metadata_size
> usr/sbin/cache_dump
> usr/sbin/cache_check
> usr/sbin/pdata_tools
> lib/modules/4.10.0-40-generic/kernel/drivers/md/dm-cache-cleaner.ko
> lib/modules/4.10.0-40-generic/kernel/drivers/md/dm-cache-smq.ko
> lib/modules/4.10.0-40-generic/kernel/drivers/md/dm-cache.ko
>
> Or similar.
>
> I assume you will be running this from your installed system. So you
> have only one chance to get it right before you need to reboot into the
> live or rescue environment.

Yeah. That's what I've encountered...

>
> If you end up on an initrd prompt (busybox) because your system doesn't
> boot, try to run:
>
>    vgchange -ay
>    vgchange -ay
>
>    exit
>
> To continue booting.


Volker


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: LVM: How to access a foreign volume group

Volker Wysk
In reply to this post by Xen
Am Sonntag, 10. Dezember 2017, 08:36:34 CET schrieb Xen:

> Volker Wysk schreef op 10-12-2017 2:38:
> > Am Samstag, 9. Dezember 2017, 22:08:54 CET schrieb Xen:
> >> Volker Wysk schreef op 04-12-2017 2:03:
> >> > Hi!
> >> >
> >> > I'm about to set up an LVM cache (for my encrypted root file system).
> >>
> >> Also I assume you will also encrypt your cache volumes?
> >>
> >> This is not easy.
> >>
> >> I don't think the initrd by default opens non-root encrypted thingies.
> >>
> >> So you would need these steps:
> >>
> >> (... ... ...)
> >
> > I'm afraid, your help goes to waste here, since I've already got it
> > running, in a virtual machine. It wasn't easy, but I've managed to set
> > it up. Your steps look familiar.
> >
> > The reason I'm worrying how to restore the system from backup, is
> > because something might go wrong, when I apply the procedure to my
> > production system, and I won't be able to boot it any longer. I want
> > to know how to recover _before_ I break my machine. I've also looked
> > at GRUB's recovey mode.
>
> Oh well then I apologize, it just seemed to me a person capable of
> setting this up would also know how to use a Live DVD ;-).
>
> In a live DVD there is nothing more to do than install
> thin-provisioning-tools and running cryptsetup open. That's it.

I'm using a live USB-Stick, no problem with that.

I've forgotten the cryptsetup step, see my other message, but now it works. I can access the root filesystem.

The work doesn't end with booting the recovery system, and mounting the root filesystem of the production system. I also need to know how to recover from the backup. This has been answered in another message. Doesn't look too difficult to me, but you must watch out.


Cheers
Volker


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

(solved) Re: LVM: How to access a foreign volume group

Volker Wysk
In reply to this post by Volker Wysk
Hi

You can either use a recovery system without encryption and without LVM, or rename its volume-group name.  You just need to do "cryptsetup open...", and the volume group of the system to be recovered will automatically emerge.

In case of two encrypted-LVM-volumes, you need to rename the vg name of the recovery system. You should *not* unlock the system-to-be-recovered yet.

Xen has made it clear how to rename the volume group:

   vgrename kubuntu-vg rescue-vg

   Then you must verify that /etc/fstab in the RESCUE system contains no
   references to "kubuntu-vg" and you might also have to rerun
   "update-grub" if you want to keep it this way.


Bye,
Volker


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: (solved) Re: LVM: How to access a foreign volume group

Volker Wysk
Am Sonntag, 10. Dezember 2017, 17:15:31 CET schrieb Volker Wysk:

> You can either use a recovery system without encryption and without LVM, or rename its volume-group name.  You just need to do "cryptsetup open...", and the volume group of the system to be recovered will automatically emerge.
>
> In case of two encrypted-LVM-volumes, you need to rename the vg name of the recovery system. You should *not* unlock the system-to-be-recovered yet.
>
> Xen has made it clear how to rename the volume group:
>
>    vgrename kubuntu-vg rescue-vg
>
>    Then you must verify that /etc/fstab in the RESCUE system contains no
>    references to "kubuntu-vg" and you might also have to rerun
>    "update-grub" if you want to keep it this way.
>

I think, I've been happy too soon. The "update-grub" step doesn't work. I get some message "/dev/mapper/kubuntu--vg-root not found". After that, the rescue system is unbootable. (Only a test USB stick affected on my side.)

Volker


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Xen
Reply | Threaded
Open this post in threaded view
|

Re: (solved) Re: LVM: How to access a foreign volume group

Xen
Volker Wysk schreef op 13-12-2017 9:00:

> Am Sonntag, 10. Dezember 2017, 17:15:31 CET schrieb Volker Wysk:
>> You can either use a recovery system without encryption and without
>> LVM, or rename its volume-group name.  You just need to do "cryptsetup
>> open...", and the volume group of the system to be recovered will
>> automatically emerge.
>>
>> In case of two encrypted-LVM-volumes, you need to rename the vg name
>> of the recovery system. You should *not* unlock the
>> system-to-be-recovered yet.
>>
>> Xen has made it clear how to rename the volume group:
>>
>>    vgrename kubuntu-vg rescue-vg
>>
>>    Then you must verify that /etc/fstab in the RESCUE system contains
>> no
>>    references to "kubuntu-vg" and you might also have to rerun
>>    "update-grub" if you want to keep it this way.
>>
>
> I think, I've been happy too soon. The "update-grub" step doesn't
> work. I get some message "/dev/mapper/kubuntu--vg-root not found".
> After that, the rescue system is unbootable. (Only a test USB stick
> affected on my side.)

Oh sorry.

It is easy to forget those complexities...

So the system was booted with /dev/mapper/kubuntu--vg-root and now
you've changed it and it can't find it anymore.

2 solutions that I can think of:

1) ln -s /dev/mapper/rescue--vg-root /dev/mapper/kubuntu--vg-root

2) use a Live DVD/stick to rename the VG first before you boot the
rescue system.

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users