Restoring backups and effect on GRUB

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Restoring backups and effect on GRUB

Kevin O'Gorman
I'm working on backup scripts, and want to make sure I understand the consequences of what I'm doing.  In particular, I have still not decided how I'm going to back up my / partition: tar/dar or dd,
and I'm wondering what effect there is on GRUB if a partition is moved, resized, or restored from a  tar/dar archive, because I'm thinking some of the above operations would change where the kernel is.  I'm unclear about how GRUB locates the kernel and any other boot files, and whether it can follow the directory structure or if instead it knows the LBA blocks where the files were originally stored.

Aside: most of the documentation addresses the benefits of some program without always explaining any details of how those benefits are achieved.  I work best when I not only know what's going to happen but also know why and how that happens.

Anyone have a clue?

++ kevin
--
Kevin O'Gorman
#define QUESTION ((bb) || (!bb))   /* Shakespeare */

Please consider the environment before printing this email.


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Restoring backups and effect on GRUB

Liam Proven
On Sat, 27 Jul 2019 at 22:17, Kevin O'Gorman <[hidden email]> wrote:
>
> Anyone have a clue?

I don't back up my OS. I should reinstall more often than I already do, in fact.

I am not saying it's a bad idea to do so. It's a good idea.

Thoughts...

*Don't* try to back up your boot sector. If you need to restore your
backup, boot off a life medium and reinstall GRUB. It's easy, and
because Windows often nukes the Linux bootsector, there's lots of info
online.

``tar'' is a very old-fashioned and low-level tool, for whole system
backups. ``dd'' even more so. ``dar'' looks a bit better but I suggest
at least evaluating higher-level tools such as AMANDA.

--
Liam Proven - Profile: https://about.me/liamproven
Email: [hidden email] - Google Mail/Hangouts/Plus: [hidden email]
Twitter/Facebook/Flickr: lproven - Skype/LinkedIn: liamproven
UK: +44 7939-087884 - ČR (+ WhatsApp/Telegram/Signal): +420 702 829 053

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Restoring backups and effect on GRUB

Ian Bruntlett
Hi,

On Sun, 28 Jul 2019 at 14:32, Liam Proven <[hidden email]> wrote:
On Sat, 27 Jul 2019 at 22:17, Kevin O'Gorman <[hidden email]> wrote:
>
> Anyone have a clue?

I have a shell script that I use to back up my personal data with tar. As for the O.S., I would reinstall it rather than restore it from backup.

HTH,


Ian

--
-- ACCU - Professionalism in programming - http://www.accu.org
-- My writing - https://sites.google.com/site/ianbruntlett/


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Restoring backups and effect on GRUB

Kevin O'Gorman
Thanks, for the ideas.  I'm still considering options.

I back up the OS because of the large number of additional packages I've installed.  I don't always remember which ones, but it could be awkward to discover one wasn't restored....

I do fresh installs instead of dist-upgrades, though, so it doesn't stay too stale...

++ kevin


On Mon, Jul 29, 2019 at 4:19 AM Ian Bruntlett <[hidden email]> wrote:
Hi,

On Sun, 28 Jul 2019 at 14:32, Liam Proven <[hidden email]> wrote:
On Sat, 27 Jul 2019 at 22:17, Kevin O'Gorman <[hidden email]> wrote:
>
> Anyone have a clue?

I have a shell script that I use to back up my personal data with tar. As for the O.S., I would reinstall it rather than restore it from backup.

HTH,


Ian

--
-- ACCU - Professionalism in programming - http://www.accu.org
-- My writing - https://sites.google.com/site/ianbruntlett/

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users


--
Kevin O'Gorman
#define QUESTION ((bb) || (!bb))   /* Shakespeare */

Please consider the environment before printing this email.


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Restoring backups and effect on GRUB

Rashkae-2
In reply to this post by Kevin O'Gorman
On 2019-07-27 4:15 p.m., Kevin O'Gorman wrote:

> I'm working on backup scripts, and want to make sure I understand the
> consequences of what I'm doing.  In particular, I have still not decided
> how I'm going to back up my / partition: tar/dar or dd,
> and I'm wondering what effect there is on GRUB if a partition is moved,
> resized, or restored from a  tar/dar archive, because I'm thinking some
> of the above operations would change where the kernel is.  I'm unclear
> about how GRUB locates the kernel and any other boot files, and whether
> it can follow the directory structure or if instead it knows the LBA
> blocks where the files were originally stored.
>

This will depend a great deal on whether or not your are booting with
Legacy Bios or EFI.  EFI makes life *much* easier, but it's taking a
long time for people to leave their comfort zone... We've been using
BIOS for a very long time.


While it's certainly possible to restore your root from a tar backup and
convince grub to repair itself,  this will probably require booting from
media to fix things up.  Make sure that you at least have a copy of
Linux Rescue CD (even if you're using it from USB stick, thats fine)
that works with your system.  Also remember, that if you intend to boot
EFI, the boot media must *also* be booted in EFI mode.


One important thing to keep in mind, to get a full copy of your root
filesystem, you must re-mount it somewhere else.  It might seem
reckless, but in Linux, you *can* mount filesystems twice.  So, for
example, your rout filesystem is on /dev/sda2

mkdir /mnt/sda2
mount /dev/sda2 /mnt/sda2

Now you can back up the contents of the /mnt/sda2

The reason for this is that the root filesystem has many files, like
devices nodes under /dev that will not get backed up from / because they
are covered by udev mounted at /mnt.  There used to be some stuff in
/proc or /sys, I forget but that that seems to be gone now.

From there, manually restoring the system from such a backup can be a
bit of a technical thing.  Sorry, I just don't have a tutorial I can
whip out.  But I would suggest you just try to do it... Put in a fresh
HDD, (or use a second computer, linux can be pretty good about that,)
and restore from your backup.



--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Restoring backups and effect on GRUB

Rashkae-2
On 2019-08-05 6:00 p.m., Rashkae wrote:

> From there, manually restoring the system from such a backup can be a
> bit of a technical thing.  Sorry, I just don't have a tutorial I can
> whip out.  But I would suggest you just try to do it... Put in a fresh
> HDD, (or use a second computer, linux can be pretty good about that,)
> and restore from your backup

So, a bit more detail....

First, it's worth noting that Ubuntu CD has some great boot repair
utilities built in.  I am, sadly, not familiar with their use, abilities
or limitations, but you should probably try that first.

It's worth noting that I'm rattling all this from memory, but I've never
successfully done this on the first try without correcting some
oversight or looking up one of the commands.  This should be used as a
guide for a trial,, fix whatever mistakes I've made from experience.

But, if all else fails and you have to repair grub the hard way, booting
into a linux system from any kind of Linux live media.

As I said last post, if you intend your system to boot EFI, the live
media must also be booted through EFI.  That will depend on your system
firmware, (typically called BIOS, but technically, BIOS only applies to
the legacy, pre-EFI systems.)  Some make it very easy to figure out if
you're booting EFI or Legacy, others make it impossible.

I do not know how to adapt this to a system with root drive on LVM, I
never tried.


Once booted into a working linux environment, mount your intended root
device somewhere easy to reach, like /mnt/root.  If applicable also
mount /boot and /boot/efi under that.

Most people probably don't have a separate /boot any more, as it is
rarely necessary, but in a worst case scenario, where /, /boot and
/boot/efi are needed

mount /dev/sda3 /mnt/root
mount /dev/sda2 /mnt/root/boot
mount /dev/sda1 /mnt/root/boot/efi

Edit the /mnt/root/etc/fstab file, change any of the UUID's that need to
be updated.

Next, you have to mount the /dev, /proc and /sys from the Live system
under the new root with --rbind option

mount --rbind /proc /mnt/root/proc
mount --rbind /dev /mnt/root/dev
mount --rbind /sys /mnt/root/sys

Now that it's ready, chroot into the new target root

chroot /mnt/root /bin/bash -i

Now that you are chrooted into the target filesystem, you can run
grub-install

For a legacy boot sytem.

grub-install /dev/sda

Or for an EFI system.

grub-install --efi-directory /boot/efi

I don't remember if you have to run update-grub, or if that happens as
part of grub-install


Reboot, and if this actually worked, run update-grub again, just to be sure.




--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Restoring backups and effect on GRUB

Colin Watson
In reply to this post by Kevin O'Gorman
On Sat, Jul 27, 2019 at 01:15:05PM -0700, Kevin O'Gorman wrote:
> I'm working on backup scripts, and want to make sure I understand the
> consequences of what I'm doing.  In particular, I have still not decided
> how I'm going to back up my / partition: tar/dar or dd,
> and I'm wondering what effect there is on GRUB if a partition is moved,
> resized, or restored from a  tar/dar archive, because I'm thinking some of
> the above operations would change where the kernel is.  I'm unclear about
> how GRUB locates the kernel and any other boot files, and whether it can
> follow the directory structure or if instead it knows the LBA blocks where
> the files were originally stored.

(I'm assuming you're using GRUB 2 rather than some long-obsolete
version.)

GRUB has file system parsing code.  In general it doesn't just encode
LBA block numbers or whatever.  There is one exception in some cases: it
has to locate its own core image somehow, since that's what contains the
file system parsing code.

If you're using UEFI, then the core image lives in the EFI System
Partition (typically mounted on /boot/efi) at a specific path which the
firmware is told to boot from; you should back up /boot/efi, but
otherwise there are no hardcoded offsets in this case.

If you're using the traditional BIOS boot mode, then booting typically
starts with a tiny first-stage loader right at the start of the disk,
which encodes the starting sector of the GRUB core image.  The first
sector of the core image in turn has a list of other sectors to read.
Once the core image is running it can then do everything else by parsing
file systems properly.

You shouldn't have a problem with any of this when merely moving or
resizing a partition (unless you manage to overwrite the GRUB core image
I suppose, but as long as you steer clear of the first megabyte of the
disk you'll more than avoid that).  Restoring can involve some work,
though.  As well as the above, restoring a file system from backup may
(depending on how you do it) change its UUID, and there are some places
(including GRUB's configuration, but possibly also /etc/fstab and
/etc/initramfs-tools/conf.d/resume) where that UUID is hardcoded.


My usual practice is to fix this all up at restore time; I always
restore using a live USB stick or similar so I have plenty of tools
available.  After I restore the file system contents, let's say to /mnt,
I do this:

  for x in dev proc run sys; do mount --rbind "/$x" "/mnt/$x"; done
  # On UEFI, you'll also need to make sure that your EFI System
  # Partition is mounted on /mnt/boot/efi.
  chroot /mnt

Then in the chroot:

  # On UEFI:
  dpkg-reconfigure grub-efi-amd64
  # ... or on BIOS (do one or the other of these, not both); this one
  # may prompt you if the disk's identity changed:
  dpkg-reconfigure grub-pc

  # This is a good point to fix up any places where the old UUID was
  # hardcoded.  I usually just do this by hand because there aren't
  # many.  You can get the new UUID from "blkid" if you need it.  Then:
  update-initramfs -u

Now exit the chroot, and tidy up:

  for x in dev proc run sys; do umount -l "/mnt/$x"; done
  # On UEFI, unmount /boot/efi as well.

You can now reboot and see what happens.  If you've made a mistake, you
still have the live USB stick to hand and you can get back in to fix
things up.

While this is certainly more than zero work, it's short enough to fit on
a couple of sticky notes (I just do it from memory, though I recognise
not everyone will quite manage that ...).

Regarding file-system-level backups versus dd or similar, I still
recommend file-system-level backups: they have the massive benefit (for
me at least) that it's completely trivial to just mount the backup disk
and fish out a particular file from a particular backup version, which
is something I do a lot more frequently than a full restore since
fat-fingering a single file by accident happens a lot more often than
trashing an entire system.  And although dd-based backups would preserve
the file system UUID, they might change other aspects of the disk's
identity, so at least on BIOS systems you'd still want to reconfigure
grub-pc as above.

--
Colin Watson                                       [[hidden email]]

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Restoring backups and effect on GRUB

Colin Watson
In reply to this post by Rashkae-2
On Mon, Aug 05, 2019 at 06:00:50PM -0400, Rashkae wrote:

> One important thing to keep in mind, to get a full copy of your root
> filesystem, you must re-mount it somewhere else.  It might seem
> reckless, but in Linux, you *can* mount filesystems twice.  So, for
> example, your rout filesystem is on /dev/sda2
>
> mkdir /mnt/sda2
> mount /dev/sda2 /mnt/sda2
>
> Now you can back up the contents of the /mnt/sda2
>
> The reason for this is that the root filesystem has many files, like
> devices nodes under /dev that will not get backed up from / because they
> are covered by udev mounted at /mnt.  There used to be some stuff in
> /proc or /sys, I forget but that that seems to be gone now.

For the record, none of the stuff that's overmounted in that way can
possibly have been relevant to anything for quite a few years; the
initramfs mounts /dev (etc.) over the top before anything would see
them.

In this case I think you're remembering something that might have once
been necessary a very long time ago, but there's no need to perpetuate
it these days.

--
Colin Watson                                       [[hidden email]]

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Restoring backups and effect on GRUB

Rashkae-2
On 2019-08-05 8:15 p.m., Colin Watson wrote:

>
> In this case I think you're remembering something that might have once
> been necessary a very long time ago, but there's no need to perpetuate
> it these days.
>


This is quite possibly, or even probably true.

There was an issue with either Ubuntu 14.04 or 16.04 with regards to
some obscure sub-directory in either /proc or /sys that caused very
early boot issues if it was missing... But I don't know how important
preserving /dev nodes are.  They do still exist.



--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Restoring backups and effect on GRUB

Robert Heller
In reply to this post by Colin Watson
At Tue, 6 Aug 2019 01:15:03 +0100 "Ubuntu user technical support,  not for general discussions" <[hidden email]> wrote:

>
> On Mon, Aug 05, 2019 at 06:00:50PM -0400, Rashkae wrote:
> > One important thing to keep in mind, to get a full copy of your root
> > filesystem, you must re-mount it somewhere else.  It might seem
> > reckless, but in Linux, you *can* mount filesystems twice.  So, for
> > example, your rout filesystem is on /dev/sda2
> >
> > mkdir /mnt/sda2
> > mount /dev/sda2 /mnt/sda2
> >
> > Now you can back up the contents of the /mnt/sda2
> >
> > The reason for this is that the root filesystem has many files, like
> > devices nodes under /dev that will not get backed up from / because they
> > are covered by udev mounted at /mnt.  There used to be some stuff in
> > /proc or /sys, I forget but that that seems to be gone now.
>
> For the record, none of the stuff that's overmounted in that way can
> possibly have been relevant to anything for quite a few years; the
> initramfs mounts /dev (etc.) over the top before anything would see
> them.
>
> In this case I think you're remembering something that might have once
> been necessary a very long time ago, but there's no need to perpetuate
> it these days.

Note: all of these "special" directories are on a different file system from
the *files* on /. /dev is a RAMDISK these days (population dynamically by udev
and friends) and /proc and /sys and so on are on special file systems on
special kernel "devices". Doing a backup/restore of / with "--one-file-system"
(or equivalent) won't touch those files.

>

--
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
[hidden email]       -- Webhosting Services
                                                                                                     

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Restoring backups and effect on GRUB

Colin Watson
In reply to this post by Rashkae-2
On Mon, Aug 05, 2019 at 08:37:17PM -0400, Rashkae wrote:
> There was an issue with either Ubuntu 14.04 or 16.04 with regards to
> some obscure sub-directory in either /proc or /sys that caused very
> early boot issues if it was missing... But I don't know how important
> preserving /dev nodes are.  They do still exist.

Yeah, I think there are a few that may still exist on the root
filesystem for one reason or another, but that's basically an accident
that it hasn't been worth anyone's time to clean up.  The initramfs puts
a tmpfs mount on top of the root filesystem's /dev before doing anything
interesting there.

I know things are OK without anything in the root filesystem's /dev
because my laptop has an empty /dev even under the tmpfs mount (so it
may be inconsistent across installations, perhaps depending on exactly
when you installed your system and with exactly which class of image).
Similarly, /proc and /sys on my laptop are empty even under the mounts.
Everything's fine.

--
Colin Watson                                       [[hidden email]]

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: Restoring backups and effect on GRUB

Liam Proven
In reply to this post by Rashkae-2
On Tue, 6 Aug 2019 at 00:03, Rashkae <[hidden email]> wrote:
>
> Legacy Bios or EFI.  EFI makes life *much* easier

:-o

*Boggle*

I really could not disagree with this statement more.

As an example, in April I spent TWO DAYS trying to get a new Dell
Latitude working for a newly-hired colleague. I failed, I got
colleagues in to consult, they failed. In the end I went to in-house
tech-support who took half a day, but did get the machine to boot
Linux.

I found _three_ different root directories on it afterwards. They
replaced mine, _twice_, in an effort to get it booting.

Note: I work for a Linux vendor and this was a machine from our
corporate compatibility list. (Our parent company was at that time a
Windows tools vendor.)

My own machine, a high-end Dell desktop, I inherited from a colleague.
He got it when his previous workstation died. He spent days trying to
get it to boot Linux from its hard disk. He failed and in the end use
a USB key with the kernel on, permanently.

I failed too. I spent days on it too.

I tried Ubuntu, Fedora and other distros. Nothing could boot.

In the end, I put Windows back on it, then dual-booted. That worked.
I'm typing on it now. It triple boots Windows, stable release and
rolling releases and it's solid now.

My girlfriend has a Lenovo desktop. It came with Win10. It is
dog-slow. I put Mint 19 on it. It's massively quicker but it won't
boot or even show GRUB unless you press F12 to bring up the boot menu
first. She always forgets so she only runs Windows.

UEFI is a catastrophic nightmare of poorly-designed, half-implemented,
under-specced bloatware. It is not merely a disaster, it is a cluster
of disasters.

It is also, IMHO, Microsoft's latest effort to destabilise and
marginalise Linux and FOSS OSes except for the big enterprise ones who
are willing to pay and sign contracts to get their kernels signed and
approved. And it's a very successful one, notwithstanding that it's
illegal, anti-competitive, dishonest and treacherous.

I am learning my way through booting FOSS OSes via UEFI but I
_despise_ it. It is a strong driver to me to move off modern x86
machines -- especially bloody Dell kit -- and restrict myself to older
Apple and Lenovo hardware as until there are good enough ARM machines,
or we can finally replace Unix, or climate catastrophe destroys our
civilisation and the entire thing becomes a moot point.

--
Liam Proven - Profile: https://about.me/liamproven
Email: [hidden email] - Google Mail/Hangouts/Plus: [hidden email]
Twitter/Facebook/Flickr: lproven - Skype/LinkedIn: liamproven
UK: +44 7939-087884 - ČR (+ WhatsApp/Telegram/Signal): +420 702 829 053

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users