How to further handle Openssl 1.1.1 in Bionic?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

How to further handle Openssl 1.1.1 in Bionic?

Christian Ehrhardt
Hi,
in recent weeks since [1][2] there were quite some bugs related to
rebuilds or feature requests.
Those kind of issues seemed to be partially expected quoting the bugs SRU text:

"OpenSSL 1.1.1 is ABI/API compatible with 1.1.0, however some software
is sensitive to the negotiation handshake and may either need
patches/improvements or clamp-down to maximum v1.2."

Along the SRU some packages were initially identified needing patches
to either fully support (if doable in an SRU scope) or clamp-down to
TLSv1.2 and got such changes.

But since then there is a set of bugs [3] coming up for either
a) "could you also enable TLSv1.3 in package ..."
b) "Openssl 1.1.1 broke package ..."

And while some of those seem to be "just work" e.g. a rebuild or small
patch to enable/disable TLSv1.3 others run into interesting issues as
openssl has changed more than jsut adding TLSv1.3.

A good example is a bug that was expected to just be a rebuild [4]. We
have realized there can be subtle effects causing regressions. In the
particular example it seems that a rebuild not only enabled TLSv1.3,
but also bumped the minimum dh key size to 2k [5] which in turn breaks
some older clients and therefore is a no-no from an SRU perspective.
The bug [4] currently is assigned to ubuntu-security team for guidance
on this - do we want/need this and accept the regression it causes or
do we want/need to "clamp-down" the dh key size as well?

But this is a generic question - not only in the context of haproxy for [4].
If formerly only "clamping down for TLSv1.2" was considered, do we
need to revisit all packages for DH key size as well? ...

Even if we don't do anything today, a security update tomorrow might
force us to rebuild and trigger this kind of issues for 'potentially'
all dependencies of openssl.
We already have seen quite some of them being actual regressions in
our LTS which is concerning for the potential estimated number of
unreported cases.

And this is what this mail is about, as more than just bug-triagers
and the security-team might have a say and an opinion about it that
should be heard. Questions are:
- Are other packages known to likely could be affected by that?
- How was that planned from the POV of the openssl upload?
  What are we expected to do in the case of [4] or any similar issue
that we find later on?
- Do we need to analyze all packages rebuilt since openssl 1.1.1 for
such effects?
...

If you are involved or have context expertise please help to clarify
the questions above for the current issue [4] but also in general.
If not I'd still ask everyone to speak up if you know or have seen
related issues.
Triagers can use the tag "bionic-openssl-1.1" to help tracking those bugs.

[1]: https://launchpad.net/ubuntu/+source/openssl/1.1.1-1ubuntu2.1~18.04.1
[2]: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1797386
[3]: https://bugs.launchpad.net/bugs/+bugs?field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.tag=bionic-openssl-1.1&orderby=status&start=0
[4]: https://bugs.launchpad.net/ubuntu/bionic/+source/haproxy/+bug/1841936
[5]: https://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-4000.html

--
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: How to further handle Openssl 1.1.1 in Bionic?

Dimitri John Ledkov
On Wed, 9 Oct 2019 at 17:41, Christian Ehrhardt
<[hidden email]> wrote:

>
> Hi,
> in recent weeks since [1][2] there were quite some bugs related to
> rebuilds or feature requests.
> Those kind of issues seemed to be partially expected quoting the bugs SRU text:
>
> "OpenSSL 1.1.1 is ABI/API compatible with 1.1.0, however some software
> is sensitive to the negotiation handshake and may either need
> patches/improvements or clamp-down to maximum v1.2."
>
> Along the SRU some packages were initially identified needing patches
> to either fully support (if doable in an SRU scope) or clamp-down to
> TLSv1.2 and got such changes.
>
> But since then there is a set of bugs [3] coming up for either
> a) "could you also enable TLSv1.3 in package ..."
> b) "Openssl 1.1.1 broke package ..."
>

There is no generic guidance we can give on what to do with these, as
they need to be on a case by case basis.

I have submitted a few rebuilds were clearly the code is sensitive to
openssl used at built time and picks up libssl1.1 >= 1.1.1 shared
library dependency based on symbols used, and those got rejected by
the SRU team.

In general, the point of openssl 1.1.1 SRU was to make openssl
supportable (maintainable, certifiable) over the Bionic timeframe. It
was not to enable TLSv1.3 across the board.

We did call the risk of connectivity issues. And those need to be
handled on a case by case basis. SNI was explicitly called out for,
and yes we had to fix about a dozen packages to do that. Arguably
things should have been using SNI for years now, and it is a security
improvement to verify, enforce, and use SNI properly. Thus when issue
is w.r.t. SNI patching to support SNI is the correct course of action
and usually simple to do.

So far connectivity issues have been minor compared with when we
enabled TLSv1.2, then had to disable it by default (but keep
compiled), then enable it back in. And these days, we are luckly
enough to have openssl.cnf system-wide way to set MaxProtocol=TLSv1.2
to prevent TLSv1.3 based connectivity issues locally.



> And while some of those seem to be "just work" e.g. a rebuild or small
> patch to enable/disable TLSv1.3 others run into interesting issues as
> openssl has changed more than jsut adding TLSv1.3.
>
> A good example is a bug that was expected to just be a rebuild [4]. We
> have realized there can be subtle effects causing regressions. In the
> particular example it seems that a rebuild not only enabled TLSv1.3,
> but also bumped the minimum dh key size to 2k [5] which in turn breaks
> some older clients and therefore is a no-no from an SRU perspective.

That's not quite correct assessment of things. We will break people
and will trade connectivity for better security. That's why we have
disabled SSLv3, mitigated poodle attacks, etc. We will continue to
require entropy, and higher key sizes, and better key-exchange
algorithms as we go along. And sometimes that includes security
updates, which SRUs build on top of. The change-effect you describe is
due to a security update of openssl, which trumps SRUs. OpenSSL 1.1.0
& 1.1.1 have raised a set of minimum key size requirements, mostly
breaking all the test-suites with pre-generated tiny keys, but some
real users too.

As a local configuration, I believe one can lower OpenSSL security
requirements by setting CipherString = DEFAULT@SECLEVEL=0 which will
bring down requirements down to like pre 1.0.2, but that should only
done as a local site configuration, and clients haunted down and
upgraded/fixed.

For the HAPROXY, it would be nice to check if CipherString =
DEFAULT@SECLEVEL=0 "helps" to bring down dh sizes to less secure ones,
and if smaller dh sizes are pre-computed/available still. I would only
call this is a regression, if there really is no way to use smaller dh
sizes and if they are stopped being available at all.

IMHO the haproxy SRU should not have been accepted, should now be
tagged proposed-regression and removed from bionic-proposed. Which is
a normal SRU process, and retry later. Or like discover some CVE and
ask security team to deal with the rebuild =)))))

> The bug [4] currently is assigned to ubuntu-security team for guidance
> on this - do we want/need this and accept the regression it causes or
> do we want/need to "clamp-down" the dh key size as well?
>

NAK

> But this is a generic question - not only in the context of haproxy for [4].
> If formerly only "clamping down for TLSv1.2" was considered, do we
> need to revisit all packages for DH key size as well? ...
>

NAK

> Even if we don't do anything today, a security update tomorrow might
> force us to rebuild and trigger this kind of issues for 'potentially'
> all dependencies of openssl.
> We already have seen quite some of them being actual regressions in
> our LTS which is concerning for the potential estimated number of
> unreported cases.
>

That was a known risk, that's why we are choosing not to blindly
rebuild things, especially at the far tail in universe.

This openssl sru in bionic has uncovered broken packages, which were
broken in cosmic, disco and eoan, despite shipping 1.1.1 there. It
does indicate that the long tail of universe packages has less than
adequate testing & autopkgtest & users in place.

> And this is what this mail is about, as more than just bug-triagers
> and the security-team might have a say and an opinion about it that
> should be heard. Questions are:
> - Are other packages known to likely could be affected by that?
> - How was that planned from the POV of the openssl upload?
>   What are we expected to do in the case of [4] or any similar issue
> that we find later on?
> - Do we need to analyze all packages rebuilt since openssl 1.1.1 for
> such effects?
> ...
>
> If you are involved or have context expertise please help to clarify
> the questions above for the current issue [4] but also in general.
> If not I'd still ask everyone to speak up if you know or have seen
> related issues.
> Triagers can use the tag "bionic-openssl-1.1" to help tracking those bugs.
>

The tag is a nice one, as it is expected for the number of issues that
do come up to die down in frequency, and each case at this point will
be unique enough requiring manual review / plan of action.


> [1]: https://launchpad.net/ubuntu/+source/openssl/1.1.1-1ubuntu2.1~18.04.1
> [2]: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1797386
> [3]: https://bugs.launchpad.net/bugs/+bugs?field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.tag=bionic-openssl-1.1&orderby=status&start=0
> [4]: https://bugs.launchpad.net/ubuntu/bionic/+source/haproxy/+bug/1841936
> [5]: https://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-4000.html
>
> --
> Christian Ehrhardt
> Software Engineer, Ubuntu Server

*Staff  ;-)

--
Regards,

Dimitri.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: How to further handle Openssl 1.1.1 in Bionic?

Christian Ehrhardt
On Thu, Oct 10, 2019 at 12:03 PM Dimitri John Ledkov <[hidden email]> wrote:

>
> On Wed, 9 Oct 2019 at 17:41, Christian Ehrhardt
> <[hidden email]> wrote:
> >
> > Hi,
> > in recent weeks since [1][2] there were quite some bugs related to
> > rebuilds or feature requests.
> > Those kind of issues seemed to be partially expected quoting the bugs SRU text:
> >
> > "OpenSSL 1.1.1 is ABI/API compatible with 1.1.0, however some software
> > is sensitive to the negotiation handshake and may either need
> > patches/improvements or clamp-down to maximum v1.2."
> >
> > Along the SRU some packages were initially identified needing patches
> > to either fully support (if doable in an SRU scope) or clamp-down to
> > TLSv1.2 and got such changes.
> >
> > But since then there is a set of bugs [3] coming up for either
> > a) "could you also enable TLSv1.3 in package ..."
> > b) "Openssl 1.1.1 broke package ..."
> >
>
> There is no generic guidance we can give on what to do with these, as
> they need to be on a case by case basis.
>
> I have submitted a few rebuilds were clearly the code is sensitive to
> openssl used at built time and picks up libssl1.1 >= 1.1.1 shared
> library dependency based on symbols used, and those got rejected by
> the SRU team.
>
> In general, the point of openssl 1.1.1 SRU was to make openssl
> supportable (maintainable, certifiable) over the Bionic timeframe. It
> was not to enable TLSv1.3 across the board.

I remember we talked about the reasoning before, thanks for the
detailed response and filling in the gaps that I had and those that I
might have forgotten :-)

> We did call the risk of connectivity issues. And those need to be
> handled on a case by case basis. SNI was explicitly called out for,
> and yes we had to fix about a dozen packages to do that. Arguably
> things should have been using SNI for years now, and it is a security
> improvement to verify, enforce, and use SNI properly. Thus when issue
> is w.r.t. SNI patching to support SNI is the correct course of action
> and usually simple to do.
>
> So far connectivity issues have been minor compared with when we
> enabled TLSv1.2, then had to disable it by default (but keep
> compiled), then enable it back in. And these days, we are luckly
> enough to have openssl.cnf system-wide way to set MaxProtocol=TLSv1.2
> to prevent TLSv1.3 based connectivity issues locally.
>
>
>
> > And while some of those seem to be "just work" e.g. a rebuild or small
> > patch to enable/disable TLSv1.3 others run into interesting issues as
> > openssl has changed more than jsut adding TLSv1.3.
> >
> > A good example is a bug that was expected to just be a rebuild [4]. We
> > have realized there can be subtle effects causing regressions. In the
> > particular example it seems that a rebuild not only enabled TLSv1.3,
> > but also bumped the minimum dh key size to 2k [5] which in turn breaks
> > some older clients and therefore is a no-no from an SRU perspective.
>
> That's not quite correct assessment of things. We will break people
> and will trade connectivity for better security. That's why we have
> disabled SSLv3, mitigated poodle attacks, etc. We will continue to
> require entropy, and higher key sizes, and better key-exchange
> algorithms as we go along. And sometimes that includes security
> updates, which SRUs build on top of. The change-effect you describe is
> due to a security update of openssl, which trumps SRUs. OpenSSL 1.1.0
> & 1.1.1 have raised a set of minimum key size requirements, mostly
> breaking all the test-suites with pre-generated tiny keys, but some
> real users too.
>
> As a local configuration, I believe one can lower OpenSSL security
> requirements by setting CipherString = DEFAULT@SECLEVEL=0 which will
> bring down requirements down to like pre 1.0.2, but that should only
> done as a local site configuration, and clients haunted down and
> upgraded/fixed.

Great suggestion, I'll give this a try somewhen later.

> For the HAPROXY, it would be nice to check if CipherString =
> DEFAULT@SECLEVEL=0 "helps" to bring down dh sizes to less secure ones,
> and if smaller dh sizes are pre-computed/available still. I would only
> call this is a regression, if there really is no way to use smaller dh
> sizes and if they are stopped being available at all.
>
> IMHO the haproxy SRU should not have been accepted, should now be
> tagged proposed-regression and removed from bionic-proposed. Which is
> a normal SRU process, and retry later. Or like discover some CVE and
> ask security team to deal with the rebuild =)))))
>
> > The bug [4] currently is assigned to ubuntu-security team for guidance
> > on this - do we want/need this and accept the regression it causes or
> > do we want/need to "clamp-down" the dh key size as well?
> >
>
> NAK
>
> > But this is a generic question - not only in the context of haproxy for [4].
> > If formerly only "clamping down for TLSv1.2" was considered, do we
> > need to revisit all packages for DH key size as well? ...
> >
>
> NAK
>
> > Even if we don't do anything today, a security update tomorrow might
> > force us to rebuild and trigger this kind of issues for 'potentially'
> > all dependencies of openssl.
> > We already have seen quite some of them being actual regressions in
> > our LTS which is concerning for the potential estimated number of
> > unreported cases.
> >
>
> That was a known risk, that's why we are choosing not to blindly
> rebuild things, especially at the far tail in universe.
>
> This openssl sru in bionic has uncovered broken packages, which were
> broken in cosmic, disco and eoan, despite shipping 1.1.1 there. It
> does indicate that the long tail of universe packages has less than
> adequate testing & autopkgtest & users in place.
>
> > And this is what this mail is about, as more than just bug-triagers
> > and the security-team might have a say and an opinion about it that
> > should be heard. Questions are:
> > - Are other packages known to likely could be affected by that?
> > - How was that planned from the POV of the openssl upload?
> >   What are we expected to do in the case of [4] or any similar issue
> > that we find later on?
> > - Do we need to analyze all packages rebuilt since openssl 1.1.1 for
> > such effects?
> > ...
> >
> > If you are involved or have context expertise please help to clarify
> > the questions above for the current issue [4] but also in general.
> > If not I'd still ask everyone to speak up if you know or have seen
> > related issues.
> > Triagers can use the tag "bionic-openssl-1.1" to help tracking those bugs.
> >
>
> The tag is a nice one, as it is expected for the number of issues that
> do come up to die down in frequency, and each case at this point will
> be unique enough requiring manual review / plan of action.
>
>
> > [1]: https://launchpad.net/ubuntu/+source/openssl/1.1.1-1ubuntu2.1~18.04.1
> > [2]: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1797386
> > [3]: https://bugs.launchpad.net/bugs/+bugs?field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.tag=bionic-openssl-1.1&orderby=status&start=0
> > [4]: https://bugs.launchpad.net/ubuntu/bionic/+source/haproxy/+bug/1841936
> > [5]: https://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-4000.html
> >
> > --
> > Christian Ehrhardt
> > Software Engineer, Ubuntu Server
>
> *Staff  ;-)
>
> --
> Regards,
>
> Dimitri.



--
Christian Ehrhardt
Staff Engineer, Ubuntu Server
Canonical Ltd

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: How to further handle Openssl 1.1.1 in Bionic?

Robie Basak-4
In reply to this post by Dimitri John Ledkov
On Thu, Oct 10, 2019 at 11:03:43AM +0100, Dimitri John Ledkov wrote:

> That's not quite correct assessment of things. We will break people
> and will trade connectivity for better security. That's why we have
> disabled SSLv3, mitigated poodle attacks, etc. We will continue to
> require entropy, and higher key sizes, and better key-exchange
> algorithms as we go along. And sometimes that includes security
> updates, which SRUs build on top of. The change-effect you describe is
> due to a security update of openssl, which trumps SRUs. OpenSSL 1.1.0
> & 1.1.1 have raised a set of minimum key size requirements, mostly
> breaking all the test-suites with pre-generated tiny keys, but some
> real users too.
>
> As a local configuration, I believe one can lower OpenSSL security
> requirements by setting CipherString = DEFAULT@SECLEVEL=0 which will
> bring down requirements down to like pre 1.0.2, but that should only
> done as a local site configuration, and clients haunted down and
> upgraded/fixed.
This is useful to know, thanks.

Is there any place we're maintaining documentation on this? It would be
handy to be able to point affected users to somewhere with an
explanation of what we're changing and why, with suggestions for
workarounds.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (836 bytes) Download Attachment