Some time around the 15.04 release, a policy change was made to quit making some logging persistent by default.
A number of users did not realize there was a policy change until they went to debug something that happened on a previous boot, only to find the logs were missing. Some of these user responses are captured in the bug report below [1], which prompted this policy discussion. The change was made because of the introduction of systemd and the introduction of the `systemd journal` in addition to the existing `rsyslog`. The concern about making the systemd journal persistent by default is that some logs could end up duplicated between the systemd journald and rsyslog, along with disk space and performance concerns of the additional logging. On the other hand, having systemd journal logs persist seems to be a safer option: It a malicious app could cause or entice a reboot, it could erase logs of it's earlier activity. Also, deleting key logs at shutdown breaks a decades-log precedent of system logs persisting through reboots. The compromise option seems to be make the systemd journal persistent by default, but minimize the amount of logging that is sent to both rsyslog and systemd to mitigate resource considerations of duplicate logging. So the policy question here is: should the systemd journal be persistent by default? Mark -- ubuntu-devel mailing list [hidden email] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
I fully understand the need to minimize redundancy and performance degradation that this incurs, which you don't want on every desktop or embedded system. But for production or test servers, this could prove invaluable (I work in a high volume system validation lab where system crashes are the norm). My suggestion (at least for existing releases) is to document the process for enabling persistent logging from the commandline, as this will allow sysadmins to easily implement it for post reboot reviews on a selective basis. Having it documented in a central location that is easily searched in google helps new sysadmins that would otherwise have to wade through tons of documentation and google searches of irrelevant information (sorry, not volunteering for this - too busy). For 18.04, maybe a separate package or a package install prompt during installation (similar to unattended-updates) could be implemented. Just my suggestion. Tobin Davis, Lab Engineer Intel Corp On Tue, Nov 7, 2017 at 12:11 PM, Mark Stosberg <[hidden email]> wrote:
-- ubuntu-devel mailing list [hidden email] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
In reply to this post by Mark Stosberg
Hi Mark,
On Tue, Nov 07, 2017 at 08:11:27PM +0000, Mark Stosberg wrote: > Some time around the 15.04 release, a policy change was made to quit > making some logging persistent by default. To help me understand the problem, exactly what logging was persistent previously, and isn't persistent now? Can you please give us an example? Thanks, Robie -- ubuntu-devel mailing list [hidden email] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
On Mon, Nov 13, 2017 at 2:27 AM, Robie Basak <[hidden email]> wrote: Hi Mark, Any data in systemd-journal that is not flush to /var/log/* is gone forever on power fail due to the journal being hosted on ephemeral tmpfs (/run/log/journal); # lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.5 LTS Release: 14.04 Codename: trusty root@t-nonvocational-azzie:/var/log# ls -al /var/log/ total 500 drwxrwxr-x 8 root syslog 4096 Nov 13 15:19 . drwxr-xr-x 12 root root 4096 Nov 10 20:43 .. drwxr-xr-x 2 root root 4096 Nov 10 20:43 apt -rw-r----- 1 syslog adm 2884 Nov 13 15:19 auth.log -rw-r--r-- 1 root root 7678 Nov 13 15:12 boot.log -rw-rw---- 1 root utmp 0 Nov 10 20:43 btmp -rw-r--r-- 1 syslog adm 153678 Nov 13 15:12 cloud-init.log -rw-r--r-- 1 root root 3870 Nov 13 15:12 cloud-init-output.log drwxr-xr-x 2 root root 4096 Nov 30 2016 dist-upgrade -rw-r--r-- 1 root adm 30483 Nov 13 15:12 dmesg -rw-r--r-- 1 root root 0 Nov 13 15:12 dmesg.0 drwxr-xr-x 2 root root 4096 Nov 10 20:41 fsck -rw-r----- 1 syslog adm 52797 Nov 13 15:13 kern.log drwxr-xr-x 2 landscape root 4096 Nov 13 15:12 landscape -rw-rw-r-- 1 root utmp 292292 Nov 13 15:19 lastlog -rw-r----- 1 syslog adm 55736 Nov 13 15:17 syslog -rw-r--r-- 1 root root 148851 Nov 13 15:12 udev drwxr-xr-x 2 root root 4096 May 8 2017 unattended-upgrades drwxr-xr-x 2 root root 4096 Nov 13 15:19 upstart -rw-rw-r-- 1 root utmp 5376 Nov 13 15:19 wtmp $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial ubuntu@x-undiluvial-edith:/var/log$ ls -al /var/log/ total 312 drwxrwxr-x 7 root syslog 4096 Nov 13 15:15 . drwxr-xr-x 13 root root 4096 Nov 10 14:25 .. drwxr-xr-x 2 root root 4096 Nov 10 14:24 apt -rw-r----- 1 syslog adm 3181 Nov 13 15:17 auth.log -rw------- 1 root utmp 0 Nov 10 14:24 btmp -rw-r--r-- 1 syslog adm 127616 Nov 13 15:15 cloud-init.log -rw-r--r-- 1 root root 4271 Nov 13 15:15 cloud-init-output.log drwxr-xr-x 2 root root 4096 Oct 20 10:35 dist-upgrade drwxr-xr-x 2 root root 4096 Nov 10 14:22 fsck -rw-r----- 1 syslog adm 52555 Nov 13 15:15 kern.log -rw-rw-r-- 1 root utmp 292292 Nov 13 15:16 lastlog drwxr-xr-x 2 root root 4096 Aug 23 02:06 lxd -rw-r----- 1 syslog adm 82360 Nov 13 15:17 syslog drwxr-x--- 2 root adm 4096 Sep 20 14:13 unattended-upgrades -rw-rw-r-- 1 root utmp 2688 Nov 13 15:16 wtmp boot.log includes services that start on boot, These service starts are not part of syslog; ssh for example in boot.log shows it starting (but any service may include more information here in case of failure). # grep -i openssh boot.log * Starting OpenSSH server [ OK ] $ journalctl -o short-precise --no-pager -u ssh -- Logs begin at Mon 2017-11-13 15:15:13 UTC, end at Mon 2017-11-13 15:17:01 UTC. -- Nov 13 15:15:33.816052 x-undiluvial-edith systemd[1]: Starting OpenBSD Secure Shell server... Nov 13 15:15:33.846646 x-undiluvial-edith sshd[1249]: Server listening on 0.0.0.0 port 22. Nov 13 15:15:33.847790 x-undiluvial-edith sshd[1249]: Server listening on :: port 22. Nov 13 15:15:33.848100 x-undiluvial-edith systemd[1]: Started OpenBSD Secure Shell server. Nov 13 15:16:35.015383 x-undiluvial-edith sshd[1317]: Accepted publickey for ubuntu from 10.5.0.10 port 54030 ssh2: RSA SHA256:0cuEJGbgH9aUy12t0jKluNEwLeAF4SIbvimjeM1OLWw Nov 13 15:16:35.018911 x-undiluvial-edith sshd[1317]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0) /var/log/udev is another. In general when systems fail to boot, users may not have console access but may retain power control. Issuing a shutdown or even hard stop will still preserve data that's been written to the filesystem for later inspection. With the bulk if boot and service status and output information included in the journald held in tmpfs; we lose that data unless there is some way to flush that data to persistent storage. I'm +1 for having journald be persistent by default; it certainly makes debugging boot failures much easier since one has actual logs to examine versus nothing at all.
-- ubuntu-devel mailing list [hidden email] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Hi Ryan,
On Mon, Nov 13, 2017 at 09:33:14AM -0600, Ryan Harper wrote: > On Mon, Nov 13, 2017 at 2:27 AM, Robie Basak <[hidden email]> wrote: > > > On Tue, Nov 07, 2017 at 08:11:27PM +0000, Mark Stosberg wrote: > > > Some time around the 15.04 release, a policy change was made to quit > > > making some logging persistent by default. > > > > To help me understand the problem, exactly what logging was persistent > > previously, and isn't persistent now? Can you please give us an example? > > > > Any data in systemd-journal that is not flush to /var/log/* is gone > forever on power fail due > to the journal being hosted on ephemeral tmpfs (/run/log/journal); said that some logging that was persistent is no longer persistent - in other words, that there has been a regression. So what exactly is only going to the systemd journal now (and thus not being persisted) that was being saved somewhere persistently before? -- ubuntu-devel mailing list [hidden email] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
In reply to this post by Robie Basak-4
On Tue, Nov 07, 2017 at 08:11:27PM +0000, Mark Stosberg wrote: Robie, As I understand, Upstart's logging about boot and managed services was persistent by default. From http://upstart.ubuntu.com/cookbook/ I find this: "For a standard Ubuntu Maverick (10.10) system, the output will be sent to file /var/log/daemon.log, whilst on newer Ubuntu systems such as Ubuntu Natty (11.04), the output will be directed to file /var/log/syslog." In turn, I understand that "syslog" logging has been persistent-by-default per years. It seems that with the switch to systemd as the init system, the logging that originates from the init system became not-persistent by default. The linked issue had examples of users who expected to find details about service logging or boot logging from a previous boot which was not found, when it would have existed under Upstart. Mark -- ubuntu-devel mailing list [hidden email] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
In reply to this post by Robie Basak-4
On Mon, Nov 13, 2017 at 03:50:20PM +0000, Robie Basak wrote:
> > So what exactly is only going to the systemd journal now (and thus not > being persisted) that was being saved somewhere persistently before? Hi Robie -- I think the closest corollary to "what has regressed" that you asked, is /var/log/upstart from trusty -> xenial. These files often contained early error messages from service startup that did not yet get to the point of logging to syslog. Similar messages show up in the journal. -- David Britton <[hidden email]> -- ubuntu-devel mailing list [hidden email] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
In reply to this post by Mark Stosberg
On Mon, 2017-11-13 at 16:12 +0000, Mark Stosberg wrote:
> *"For a standard Ubuntu Maverick (10.10) system, the output will be > sent to file /var/log/daemon.log, whilst on newer Ubuntu systems > such as Ubuntu Natty (11.04), the output will be directed to file > /var/log/syslog."* > > In turn, I understand that "syslog" logging has been persistent-by- > default per years. > > It seems that with the switch to systemd as the init system, the > logging that originates from the init system became not-persistent by > default. $ grep -c 'systemd\[1\]:' /var/log/syslog 39 It's still there... -- Jan Claeys -- ubuntu-devel mailing list [hidden email] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Jan, As noted in the linked bug report, there are some logs which are uniquely stored in the systemd journal, the journal is not persistent-by-default, and data loss is occurring as a default behavior. On Fri, Nov 17, 2017 at 11:21 AM Jan Claeys <[hidden email]> wrote: On Mon, 2017-11-13 at 16:12 +0000, Mark Stosberg wrote: -- ubuntu-devel mailing list [hidden email] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
On Fri, 2017-11-17 at 17:10 +0000, Mark Stosberg wrote:
> As noted in the linked bug report, there are some logs which are > uniquely stored in the systemd journal, the journal is not > persistent-by-default, and data loss is occurring as a default > behavior. AFAICT rsyslogd wasn't special-cased in the upstart days, meaning it would get killed with other upstart services in semi-random order, so it never was able to log the very last messages. It would get killed after most services that were defined by a SysV init script though. I'm not sure how much of a regression there really is; it would depend on how early rsyslogd gets killed now... If journald uses some specific configuration/dependency to always be stopped as late as possible, then maybe that can be applied to the rsyslogd unit too, so that they get killed virtually together (or even rsyslogd last, if that's possible). That could solve your issue in current releases even? -- Jan Claeys -- ubuntu-devel mailing list [hidden email] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
In reply to this post by Mark Stosberg
Hey there,
Le 07/11/2017 à 21:11, Mark Stosberg a écrit : should the systemd journal be persistent by default? Just as a follow up on that topic, Dimitri made the journal persistent by default now in bionic https://launchpad.net/ubuntu/+source/systemd/235-3ubuntu3
Cheers, Sebastien Bacher -- ubuntu-devel mailing list [hidden email] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel |
Free forum by Nabble | Edit this page |