Should Ubuntu systemd journal logs be persistent by default?

[Mark Stosberg]

Some time around the 15.04 release, a policy change was made to quit making some logging persistent by default.

A number of users did not realize there was a policy change until they went to debug something that happened on a previous boot, only to find the logs were missing. Some of these user responses are captured in the bug report below [1], which prompted this policy discussion.

The change was made because of the introduction of systemd and the introduction of the `systemd journal` in addition to the existing `rsyslog`.

The concern about making the systemd journal persistent by default is that some logs could end up duplicated between the systemd journald and rsyslog, along with disk space and performance concerns of the additional logging. 

On the other hand, having systemd journal logs persist seems to be a safer option: It a malicious app could cause or entice a reboot, it could erase logs of it's earlier activity. Also, deleting key logs at shutdown breaks a decades-log precedent of  system logs persisting through reboots. 

The compromise option seems to be make the systemd journal persistent by default, but minimize the amount of  logging that is sent to both rsyslog and systemd to mitigate resource considerations of duplicate logging.

So the policy question here is: should the systemd journal be persistent by default?

     Mark

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1618188

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

I fully understand the need to minimize redundancy and performance degradation that this incurs, which you don't want on every desktop or embedded system.  But for production or test servers, this could prove invaluable (I work in a high volume system validation lab where system crashes are the norm).

My suggestion (at least for existing releases) is to document the process for enabling persistent logging from the commandline, as this will allow sysadmins to easily implement it for post reboot reviews on a selective basis.  Having it documented in a central location that is easily searched in google helps new sysadmins that would otherwise have to wade through tons of documentation and google searches of irrelevant information (sorry, not volunteering for this - too busy).

For 18.04, maybe a separate package or a package install prompt during installation (similar to unattended-updates) could be implemented.

Just my suggestion.

Tobin Davis, Lab Engineer

Intel Corp

On Tue, Nov 7, 2017 at 12:11 PM, Mark Stosberg <[hidden email]> wrote:

Some time around the 15.04 release, a policy change was made to quit making some logging persistent by default.

A number of users did not realize there was a policy change until they went to debug something that happened on a previous boot, only to find the logs were missing. Some of these user responses are captured in the bug report below [1], which prompted this policy discussion.

The change was made because of the introduction of systemd and the introduction of the `systemd journal` in addition to the existing `rsyslog`.

The concern about making the systemd journal persistent by default is that some logs could end up duplicated between the systemd journald and rsyslog, along with disk space and performance concerns of the additional logging. 

On the other hand, having systemd journal logs persist seems to be a safer option: It a malicious app could cause or entice a reboot, it could erase logs of it's earlier activity. Also, deleting key logs at shutdown breaks a decades-log precedent of  system logs persisting through reboots. 

The compromise option seems to be make the systemd journal persistent by default, but minimize the amount of  logging that is sent to both rsyslog and systemd to mitigate resource considerations of duplicate logging.

So the policy question here is: should the systemd journal be persistent by default?

     Mark

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1618188

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Robie Basak-4]

Hi Mark,

On Tue, Nov 07, 2017 at 08:11:27PM +0000, Mark Stosberg wrote:
> Some time around the 15.04 release, a policy change was made to quit
> making some logging persistent by default.

To help me understand the problem, exactly what logging was persistent
previously, and isn't persistent now? Can you please give us an example?

Thanks,

Robie

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Ryan Harper]

On Mon, Nov 13, 2017 at 2:27 AM, Robie Basak <[hidden email]> wrote:

Hi Mark,

On Tue, Nov 07, 2017 at 08:11:27PM +0000, Mark Stosberg wrote:
> Some time around the 15.04 release, a policy change was made to quit
> making some logging persistent by default.

To help me understand the problem, exactly what logging was persistent
previously, and isn't persistent now? Can you please give us an example?

Any data in systemd-journal that is not flush to  /var/log/* is gone forever on power fail due

to the journal being hosted on ephemeral tmpfs (/run/log/journal); 

# lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 14.04.5 LTS

Release: 14.04

Codename: trusty

root@t-nonvocational-azzie:/var/log# ls -al /var/log/

total 500

drwxrwxr-x  8 root      syslog   4096 Nov 13 15:19 .

drwxr-xr-x 12 root      root     4096 Nov 10 20:43 ..

drwxr-xr-x  2 root      root     4096 Nov 10 20:43 apt

-rw-r-----  1 syslog    adm      2884 Nov 13 15:19 auth.log

-rw-r--r--  1 root      root     7678 Nov 13 15:12 boot.log

-rw-rw----  1 root      utmp        0 Nov 10 20:43 btmp

-rw-r--r--  1 syslog    adm    153678 Nov 13 15:12 cloud-init.log

-rw-r--r--  1 root      root     3870 Nov 13 15:12 cloud-init-output.log

drwxr-xr-x  2 root      root     4096 Nov 30  2016 dist-upgrade

-rw-r--r--  1 root      adm     30483 Nov 13 15:12 dmesg

-rw-r--r--  1 root      root        0 Nov 13 15:12 dmesg.0

drwxr-xr-x  2 root      root     4096 Nov 10 20:41 fsck

-rw-r-----  1 syslog    adm     52797 Nov 13 15:13 kern.log

drwxr-xr-x  2 landscape root     4096 Nov 13 15:12 landscape

-rw-rw-r--  1 root      utmp   292292 Nov 13 15:19 lastlog

-rw-r-----  1 syslog    adm     55736 Nov 13 15:17 syslog

-rw-r--r--  1 root      root   148851 Nov 13 15:12 udev

drwxr-xr-x  2 root      root     4096 May  8  2017 unattended-upgrades

drwxr-xr-x  2 root      root     4096 Nov 13 15:19 upstart

-rw-rw-r--  1 root      utmp     5376 Nov 13 15:19 wtmp

$ lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 16.04.3 LTS

Release: 16.04

Codename: xenial

ubuntu@x-undiluvial-edith:/var/log$ ls -al /var/log/

total 312

drwxrwxr-x  7 root   syslog   4096 Nov 13 15:15 .

drwxr-xr-x 13 root   root     4096 Nov 10 14:25 ..

drwxr-xr-x  2 root   root     4096 Nov 10 14:24 apt

-rw-r-----  1 syslog adm      3181 Nov 13 15:17 auth.log

-rw-------  1 root   utmp        0 Nov 10 14:24 btmp

-rw-r--r--  1 syslog adm    127616 Nov 13 15:15 cloud-init.log

-rw-r--r--  1 root   root     4271 Nov 13 15:15 cloud-init-output.log

drwxr-xr-x  2 root   root     4096 Oct 20 10:35 dist-upgrade

drwxr-xr-x  2 root   root     4096 Nov 10 14:22 fsck

-rw-r-----  1 syslog adm     52555 Nov 13 15:15 kern.log

-rw-rw-r--  1 root   utmp   292292 Nov 13 15:16 lastlog

drwxr-xr-x  2 root   root     4096 Aug 23 02:06 lxd

-rw-r-----  1 syslog adm     82360 Nov 13 15:17 syslog

drwxr-x---  2 root   adm      4096 Sep 20 14:13 unattended-upgrades

-rw-rw-r--  1 root   utmp     2688 Nov 13 15:16 wtmp

boot.log includes services that start on boot, 

These service starts are not part of syslog;  ssh for example in boot.log shows

it starting (but any service may include more information here in case of failure).

# grep -i openssh boot.log 

 * Starting OpenSSH server                                               [ OK ]

$ journalctl -o short-precise --no-pager -u ssh

-- Logs begin at Mon 2017-11-13 15:15:13 UTC, end at Mon 2017-11-13 15:17:01 UTC. --

Nov 13 15:15:33.816052 x-undiluvial-edith systemd[1]: Starting OpenBSD Secure Shell server...

Nov 13 15:15:33.846646 x-undiluvial-edith sshd[1249]: Server listening on 0.0.0.0 port 22.

Nov 13 15:15:33.847790 x-undiluvial-edith sshd[1249]: Server listening on :: port 22.

Nov 13 15:15:33.848100 x-undiluvial-edith systemd[1]: Started OpenBSD Secure Shell server.

Nov 13 15:16:35.015383 x-undiluvial-edith sshd[1317]: Accepted publickey for ubuntu from 10.5.0.10 port 54030 ssh2: RSA SHA256:0cuEJGbgH9aUy12t0jKluNEwLeAF4SIbvimjeM1OLWw

Nov 13 15:16:35.018911 x-undiluvial-edith sshd[1317]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)

/var/log/udev is another.

In general when systems fail to boot, users may not have console access but may retain power control.

Issuing a shutdown or even hard stop will still preserve data that's been written to the filesystem for later

inspection.  With the bulk if boot and service status and output information included in the journald held in

tmpfs; we lose that data unless there is some way to flush that data to persistent storage.

I'm +1 for having journald be persistent by default; it certainly makes debugging boot failures

much easier since one has actual logs to examine versus nothing at all.

Thanks,

Robie

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Robie Basak-4]

Hi Ryan,

On Mon, Nov 13, 2017 at 09:33:14AM -0600, Ryan Harper wrote:

> On Mon, Nov 13, 2017 at 2:27 AM, Robie Basak <[hidden email]> wrote:
>
> > On Tue, Nov 07, 2017 at 08:11:27PM +0000, Mark Stosberg wrote:
> > > Some time around the 15.04 release, a policy change was made to quit
> > > making some logging persistent by default.
> >
> > To help me understand the problem, exactly what logging was persistent
> > previously, and isn't persistent now? Can you please give us an example?
> >
>
> Any data in systemd-journal that is not flush to  /var/log/* is gone
> forever on power fail due
> to the journal being hosted on ephemeral tmpfs (/run/log/journal);

I understand that part, but that doesn't really answer my question. Mark
said that some logging that was persistent is no longer persistent - in
other words, that there has been a regression.

So what exactly is only going to the systemd journal now (and thus not
being persisted) that was being saved somewhere persistently before?

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Mark Stosberg]

On Tue, Nov 07, 2017 at 08:11:27PM +0000, Mark Stosberg wrote:
> Some time around the 15.04 release, a policy change was made to quit
> making some logging persistent by default.

To help me understand the problem, exactly what logging was persistent
previously, and isn't persistent now? Can you please give us an example?

Robie,

As I understand, Upstart's logging about boot and managed services was persistent by default.  From http://upstart.ubuntu.com/cookbook/ I find this:

"For a standard Ubuntu Maverick (10.10) system, the output will be sent to file /var/log/daemon.log, whilst on newer Ubuntu systems such as Ubuntu Natty (11.04), the output will be directed to file /var/log/syslog."

In turn, I understand that "syslog" logging has been persistent-by-default per years. 

It seems that with the switch to systemd as the init system, the logging that originates from the init system became not-persistent by default.

The linked issue had examples of users who expected to find details about service logging or boot logging from a previous boot which was not found, when it would have existed under Upstart.

     Mark             

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[David Britton]

On Mon, Nov 13, 2017 at 03:50:20PM +0000, Robie Basak wrote:
>
> So what exactly is only going to the systemd journal now (and thus not
> being persisted) that was being saved somewhere persistently before?

Hi Robie --

I think the closest corollary to "what has regressed" that you asked,
is /var/log/upstart from trusty -> xenial.  These files often contained
early error messages from service startup that did not yet get to the
point of logging to syslog.

Similar messages show up in the journal.

--
David Britton <[hidden email]>

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Jan Claeys-3]

On Mon, 2017-11-13 at 16:12 +0000, Mark Stosberg wrote:

> *"For a standard Ubuntu Maverick (10.10) system, the output will be
> sent to file /var/log/daemon.log, whilst on newer Ubuntu systems
> such as Ubuntu Natty (11.04), the output will be directed to file
> /var/log/syslog."*
>
> In turn, I understand that "syslog" logging has been persistent-by-
> default per years.
>
> It seems that with the switch to systemd as the init system, the
> logging that originates from the init system became not-persistent by
> default.

    $ grep -c 'systemd\[1\]:' /var/log/syslog
    39

It's still there...


--
Jan Claeys


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Mark Stosberg]

Jan,

As noted in the linked bug report, there are some logs which are uniquely stored in the systemd journal, the journal is not persistent-by-default, and data loss is occurring as a default behavior. 

On Fri, Nov 17, 2017 at 11:21 AM Jan Claeys <[hidden email]> wrote:

On Mon, 2017-11-13 at 16:12 +0000, Mark Stosberg wrote:
> *"For a standard Ubuntu Maverick (10.10) system, the output will be
> sent to file /var/log/daemon.log, whilst on newer Ubuntu systems
> such as Ubuntu Natty (11.04), the output will be directed to file
> /var/log/syslog."*
>
> In turn, I understand that "syslog" logging has been persistent-by-
> default per years.
>
> It seems that with the switch to systemd as the init system, the
> logging that originates from the init system became not-persistent by
> default.

    $ grep -c 'systemd\[1\]:' /var/log/syslog
    39

It's still there...


--
Jan Claeys

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Jan Claeys-3]

On Fri, 2017-11-17 at 17:10 +0000, Mark Stosberg wrote:
> As noted in the linked bug report, there are some logs which are
> uniquely stored in the systemd journal, the journal is not
> persistent-by-default, and data loss is occurring as a default
> behavior.

AFAICT rsyslogd wasn't special-cased in the upstart days, meaning it
would get killed with other upstart services in semi-random order, so
it never was able to log the very last messages.  It would get killed
after most services that were defined by a SysV init script though.
I'm not sure how much of a regression there really is; it would depend
on how early rsyslogd gets killed now...


If journald uses some specific configuration/dependency to always be
stopped as late as possible, then maybe that can be applied to the
rsyslogd unit too, so that they get killed virtually together (or even
rsyslogd last, if that's possible).  That could solve your issue in
current releases even?


--
Jan Claeys


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

[Sebastien Bacher]

Hey there,

Le 07/11/2017 à 21:11, Mark Stosberg a écrit :

should the systemd journal be persistent by default?

Just as a follow up on that topic, Dimitri made the journal persistent by default now in bionic

https://launchpad.net/ubuntu/+source/systemd/235-3ubuntu3

Cheers,

Sebastien Bacher

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel