Bugs reports should include syslog warnings or not?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Bugs reports should include syslog warnings or not?

Sebastien Bacher
Hey there,

https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1738581 was raised
to my attention in a discussion about apport/e.u.c and I'm wondering if
the change is right

The report pointed out that private info have been included in a report
through JournalError.txt, and the solution applied was to change apport
to include errors level messages only and not warning.

Looking a bit a journalerror on some bugs it seems we have indeed some
components that log too much content as "warning" (gdm in that case),
but changing to "error" has been cutting out useful warnings and doesn't
seem the right fix to me nor a step in the right direction. It doesn't
also protect us of the described issue (if a program logs sensitive info
in its errors messages we are still going to send them).

I suggest that we change apport back to report warnings as well and look
at how we can better fix the privacy issue.

The xession logs are filtering on "safe" keywords, maybe one option
would be to do something similar for the journal

https://bazaar.launchpad.net/~apport-hackers/apport/trunk/view/head:/apport/hookutils.py#L517

Another thing we could/should do is to review the logs and fix programs
that are logging too much details to the journal as the warning/error
levels.

What do you think?


Cheers,
Sebastien Bacher


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Bugs reports should include syslog warnings or not?

Robie Basak-4
On Sat, Mar 17, 2018 at 06:09:25PM +0100, Sebastien Bacher wrote:
> The report pointed out that private info have been included in a report
> through JournalError.txt, and the solution applied was to change apport
> to include errors level messages only and not warning.

IMHO, not logging warning level messages is too blunt an instrument to
fix this bug. And it doesn't really fix it either - the next time it
might be that private data is leaked via an error rather than a warning.

IMHO, private information should never be leaked to logs by default, by
being obfuscated at source. An exception might be if a developer
explicitly and specifically turns on such an option having had the
opportunity to understand the consequences and take the necessary care.

I also think that while private information should of course be removed,
the log line should still be present (eg. "Sent: <private information
removed>" instead of nothing at all). Developers don't usually need to
know a specific secret, but the fact that the event happened is
sometimes very useful for debugging.

This isn't just for apport: people doing community support (such as IRC,
askubuntu.com, etc) quite reasonably encourage the pastebinning of
appropriate logs, and just doing something in apport will not fix this
underlying problem.

In this case, I don't know enough about the stack in question and I
wasn't able to gather this from reading the logs. Is the problem that
one process is setting a secret in an environment variable and another
process is "innocently" reporting an environment variable that has been
set not knowing that it is a secret? Perhaps the way the stack operates
needs to be revisited if so.

> The xession logs are filtering on "safe" keywords, maybe one option
> would be to do something similar for the journal
>
> https://bazaar.launchpad.net/~apport-hackers/apport/trunk/view/head:/apport/hookutils.py#L517
>
> Another thing we could/should do is to review the logs and fix programs
> that are logging too much details to the journal as the warning/error
> levels.

Agreed.

For example, in MySQL, we once had an edge case reported where it did
leak passwords (LP: #1574458). It was treated as an upstream bug which
got fixed. In the meantime, we SRU'd an apport workaround to amend the
known bad strings. This code is still present:
https://salsa.debian.org/mariadb-team/mysql/blob/mysql-5.7/debian/master/debian/additions/source_mysql-5.7.py#L24

I think this is a reasonable pattern to follow: treat it as a privacy
leak bug, fix the software upstream to stop logging it by default, and
distro-patch or adjust apport hooks to work around the problem until the
upstream fix arrives.

Robie

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bugs reports should include syslog warnings or not?

Jeremy Bicha-2
On Sat, Mar 17, 2018 at 7:51 PM, Robie Basak <[hidden email]> wrote:

> On Sat, Mar 17, 2018 at 06:09:25PM +0100, Sebastien Bacher wrote:
>> The report pointed out that private info have been included in a report
>> through JournalError.txt, and the solution applied was to change apport
>> to include errors level messages only and not warning.
>
> IMHO, not logging warning level messages is too blunt an instrument to
> fix this bug. And it doesn't really fix it either - the next time it
> might be that private data is leaked via an error rather than a warning.
>
> IMHO, private information should never be leaked to logs by default, by
> being obfuscated at source. An exception might be if a developer
> explicitly and specifically turns on such an option having had the
> opportunity to understand the consequences and take the necessary care.

One particular class of private info I've seen in the systemd journal
is file names of files that tracker fails to index.

File names can be very sensitive. And yet, it seems to me like it's
appropriate for tracker to log the file name as a warning.

Maybe apport should exclude tracker warnings by default for bugs that
aren't related to tracker?

Thanks,
Jeremy Bicha

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Bugs reports should include syslog warnings or not?

Robie Basak-4
On Sat, Mar 17, 2018 at 08:13:55PM -0400, Jeremy Bicha wrote:
> One particular class of private info I've seen in the systemd journal
> is file names of files that tracker fails to index.
>
> File names can be very sensitive. And yet, it seems to me like it's
> appropriate for tracker to log the file name as a warning.

The way I see it, by choosing to log, one is also choosing to make that
data public should the user share logs. Since sharing logs is something
that is typically done when asking for help on the Internet at large.

apport is only one part of this. Special casing privacy considerations
in apport, IMHO, doesn't help with any wider privacy leak when a user is
asked to share logs some other way.

I conclude that it needs to be decided in tracker upstream if that
information should be considered private or not. If it should be
private, then it shouldn't be logged by upstream by default. One way to
solve this might be to log the warning with private information not
present, but provide some other way to reveal the detail. This could be
by enabling some privacy-compromising-logging flag and requring the user
to rerun, or by storing the private information somewhere
out-of-default-band.

> Maybe apport should exclude tracker warnings by default for bugs that
> aren't related to tracker?

I have no objection to mitigating privacy concerns in apport in this way
in lieu of the proper type of fix I suggest above. In the general case I
think we absolutely should do this in the absence of an upstream fix.
But please don't exclude entire messages, as that can be confusing for
debugging; please instead leave a placeholder excluding the private
information.

In this specific case, I suppose it depends on whether we (the wider
community including upstream) decide whether or not it is a privacy
problem in this particular instance.

Robie

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bugs reports should include syslog warnings or not?

Matthew Paul Thomas
Robie Basak wrote on 19/03/18 13:47:

>
> On Sat, Mar 17, 2018 at 08:13:55PM -0400, Jeremy Bicha wrote:
>>
>> One particular class of private info I've seen in the systemd journal
>> is file names of files that tracker fails to index.
>>
>> File names can be very sensitive. And yet, it seems to me like it's
>> appropriate for tracker to log the file name as a warning.
>
> The way I see it, by choosing to log, one is also choosing to make
> that data public should the user share logs. Since sharing logs is
> something that is typically done when asking for help on the Internet
> at large.

If I understand this correctly, the logic is:

1.  People choose whether to log systemd.

2.  Those people, who choose to log systemd, know that “ubuntu-bug
    evolution” (for example) will post JournalErrors.txt publicly.

3.  Those people, who know they’re posting JournalErrors.txt publicly,
    also know that it may include confidential information.

Is that right? Because I’d be surprised if *any* of those things is true
(for more than 10% of that set of people), let alone all three.

> apport is only one part of this. Special casing privacy considerations
> in apport, IMHO, doesn't help with any wider privacy leak when a user
> is asked to share logs some other way.
>
> I conclude that it needs to be decided in tracker upstream if that
> information should be considered private or not. If it should be
> private, then it shouldn't be logged by upstream by default.
>…

This seems to assume that the main use of Ubuntu log files is posting in
public bug reports and support forums — rather than, say,
troubleshooting and system administration in corporate IT departments.
Again, I’d be surprised if that’s true.

--
mpt

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Bugs reports should include syslog warnings or not?

Robie Basak-4
On Mon, Mar 19, 2018 at 03:55:25PM +0000, Matthew Paul Thomas wrote:

> Robie Basak wrote on 19/03/18 13:47:
> > The way I see it, by choosing to log, one is also choosing to make
> > that data public should the user share logs. Since sharing logs is
> > something that is typically done when asking for help on the Internet
> > at large.
>
> If I understand this correctly, the logic is:
>
> 1.  People choose whether to log systemd.
>
> 2.  Those people, who choose to log systemd, know that “ubuntu-bug
>     evolution” (for example) will post JournalErrors.txt publicly.
>
> 3.  Those people, who know they’re posting JournalErrors.txt publicly,
>     also know that it may include confidential information.
>
> Is that right? Because I’d be surprised if *any* of those things is true
> (for more than 10% of that set of people), let alone all three.
No, I think you have the inverse sense of what I intended. I mean that
by the _developer_ choosing to write upstream code such that something
is logged, that developer is also implicitly deciding that the logs may
be made public, because that's how the ecosystem works. So upstreams
should ensure that private information is not logged by default.

> > I conclude that it needs to be decided in tracker upstream if that
> > information should be considered private or not. If it should be
> > private, then it shouldn't be logged by upstream by default.
> >…
>
> This seems to assume that the main use of Ubuntu log files is posting in
> public bug reports and support forums — rather than, say,
> troubleshooting and system administration in corporate IT departments.
> Again, I’d be surprised if that’s true.

For a privacy concern, I don't think it matters what the main use is. A
minority use that leads to a leak is still a leak that we should fix.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bugs reports should include syslog warnings or not?

Matthew Paul Thomas
Robie Basak wrote on 19/03/18 19:41:
>…> No, I think you have the inverse sense of what I intended. I mean
that> by the _developer_ choosing to write upstream code such that
something> is logged,
Ah, I see, I misinterpreted “one” as referring to the user.

>            that developer is also implicitly deciding that the logs
> may be made public, because that's how the ecosystem works. So
> upstreams should ensure that private information is not logged by
> default.
>…
>> This seems to assume that the main use of Ubuntu log files is posting
>> in public bug reports and support forums — rather than, say,
>> troubleshooting and system administration in corporate IT
>> departments. Again, I’d be surprised if that’s true.
>
> For a privacy concern, I don't think it matters what the main use is.
> A minority use that leads to a leak is still a leak that we should
> fix.

The proportion of use determines *how* it should be fixed. If many/most
uses of a log are for private troubleshooting and system administration,
then expecting every upstream developer to omit useful information when
logging — or to store “the private information somewhere
out-of-default-band” — would not be the most efficient solution.

--
mpt

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Bugs reports should include syslog warnings or not?

Brian Murray-5
In reply to this post by Sebastien Bacher
On Sat, Mar 17, 2018 at 06:09:25PM +0100, Sebastien Bacher wrote:
> Hey there,
>
> https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1738581 was raised
> to my attention in a discussion about apport/e.u.c and I'm wondering if
> the change is right

Thanks for bringing this up.

> The report pointed out that private info have been included in a report
> through JournalError.txt, and the solution applied was to change apport
> to include errors level messages only and not warning.
>
> Looking a bit a journalerror on some bugs it seems we have indeed some
> components that log too much content as "warning" (gdm in that case),
> but changing to "error" has been cutting out useful warnings and doesn't
> seem the right fix to me nor a step in the right direction. It doesn't
> also protect us of the described issue (if a program logs sensitive info
> in its errors messages we are still going to send them).
>
> I suggest that we change apport back to report warnings as well and look
> at how we can better fix the privacy issue.

I've modified apport back to include warnings but at the same time to
address the privacy issue have also changed apport to only include
JournalErrors when the report is a crash report as those reports are
private by default. So before making a crash report public be sure to
review the JournalErrors attachment for private information. And of
course you can always ask the bug reporter to run the same command,
'journalctl -b --priority=warning --lines=1000', and add that to their
regular bug reports if necessary.

--
Brian Murray

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel