Should we be reverting iptables to iptables-legacy for eoan?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Should we be reverting iptables to iptables-legacy for eoan?

Julian Andres Klode
Hi folks,

it turns out that lxd is broken by iptables now using the nft
based stuff, because lxd is still using the legacy one from
inside the snap.

This provides a terrible experience because networking in lxd
is not working at all once you enable ufw.

I'd suggest we increase the priority of iptables-legacy for eoan,
so that it is the default, and move the switch to xtables-nft-based
one to next release.

This will allow us to have working lxd networking, and gives
the lxd team some breathing room.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Should we be reverting iptables to iptables-legacy for eoan?

Stéphane Graber
Hi,

I'm strongly on the revert camp. This change landed in the release
pocket after Feature Freeze and effectively causes the majority of
software in the Ubuntu archive to rely on compatibility wrappers to
function.
Those wrappers aren't perfect match for the tools they replace,
causing failures or worse, success but leading to non-matching rules
in nft.

Due to both legacy iptables and nft functioning at the same time, we
may also now be in the very confusing situation of having some rules
loaded into nft while some older tools load directly into iptables,
causing only half the rules to be visible.

I agree that nft is the future, but this needs coordination, we need
to figure out what packages in main are using
iptables/ip6tables/ebtables and make sure that they're either
perfectly compatible with the compat wrappers OR have native nft
support.
And for software that grows direct nft support, they will need to
detect whether nft is in use rather than just check if it's supported
by the kernel, that's necessary to avoid ending up with rules in both.

For LXD specifically, we think it would take us about 3 weeks of
engineering work to sort this in a way that can work on all
distributions, properly detecting and supporting:
 - no nft present
 - nft present but old iptables used
 - nft present and used

The other problem we'll need to look into is that many of the nft
tools appear and user guides start with doing a full clear of nft
before populating it and using it, so that may cause some headaches
due to ordering as LXD needs to add some rules, obviously doesn't want
things reset underneath it. nft is actually more flexible than
iptables for updating the ruleset, but we need to ensure it's setup
properly so we don't end up with surprises.

Stéphane


On Tue, Sep 10, 2019 at 5:12 PM Julian Andres Klode
<[hidden email]> wrote:

>
> Hi folks,
>
> it turns out that lxd is broken by iptables now using the nft
> based stuff, because lxd is still using the legacy one from
> inside the snap.
>
> This provides a terrible experience because networking in lxd
> is not working at all once you enable ufw.
>
> I'd suggest we increase the priority of iptables-legacy for eoan,
> so that it is the default, and move the switch to xtables-nft-based
> one to next release.
>
> This will allow us to have working lxd networking, and gives
> the lxd team some breathing room.
>
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer                              i speak de, en
>
> --
> ubuntu-devel mailing list
> [hidden email]
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel



--
Stéphane

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: Should we be reverting iptables to iptables-legacy for eoan?

Jamie Strandboge-3
In reply to this post by Julian Andres Klode
On Tue, 10 Sep 2019, Julian Andres Klode wrote:

> Hi folks,
>
> it turns out that lxd is broken by iptables now using the nft
> based stuff, because lxd is still using the legacy one from
> inside the snap.
>
> This provides a terrible experience because networking in lxd
> is not working at all once you enable ufw.

Is this because ufw uses whatever is on the system (ie, nft default) and
lxd is using whatever is in the snap (ie, historic iptables which with
iptables 1.8 translates to 'iptables-legacy') and this means that both
backends are in use, which results in a mixed view of things?

The man pages for iptables-nft/iptables-legacy don't call this out as a
problem, but README.Debian does:

  NOTE: make sure you don't mix iptables-legacy and iptables-nft
  rulesets in the same machine at the same time just for sanity and to
  avoid unexpected behaviours in your network.

> I'd suggest we increase the priority of iptables-legacy for eoan,
> so that it is the default, and move the switch to xtables-nft-based
> one to next release.
>
> This will allow us to have working lxd networking, and gives
> the lxd team some breathing room.

This makes sense to me. Besides, there are still bugs trickling in where
the nft backend isn't acting fully compatibly anyway. However, it would
be nice to use the nft backend by default by 20.04, ie, at the opening
of the cycle.

That said, this is a very real immediate problem for snaps on 19.10+,
Buster+ and Ubuntu Core (CC'ing Samuele and Michael specifically) since
the only available bases are based on xenial and bionic and snaps can
only stage-packages from xenial and bionic, so snaps that plugs
'firewall-control' (or classic snaps) will by definition by far tend to
use 'legacy' (unless they build iptables from source). Even though the
system might have the nft backend enabled. Furthermore, the man page for
iptables-nft says that kernel >= 4.17 is required for the nft backend
(but ISTR issues in Debian with > 4.19) and its entirely possible
someone could be running a system with an older kernel with a newer user
space (eg, core20 snap with bionic kernel snap).

I'm not sure how to fix this. The only thing that seems reasonable is
for snaps to only be able to use the correct one that matches the
host/kernel. To pull that off snapd would need to examine the
capabilities and configuration of the running system then bind mount
iptables/etc from the host into runtime of the snap (the review-tools
could flag if snaps ship these binaries). Alternatively, an iptables
snapcraft part could be created that at runtime can interrogate whether
to use the nft or legacy backends (or snapd exposes which backend is in
use to the snap so it can decide).

I suspect that Michael and Samuele will want to move the snap-specific
parts of this discussion to the snapcraft forum, but +1 from me to
prefer 'legacy' for the time being in Ubuntu.

--
Jamie Strandboge             | http://www.canonical.com

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Should we be reverting iptables to iptables-legacy for eoan?

Jamie Strandboge-3
On Tue, 10 Sep 2019, Jamie Strandboge wrote:

> (but ISTR issues in Debian with > 4.19)

Sorry, meant < 4.19 (ie, 4.17 and 4.18 istr still had some issues)

--
Jamie Strandboge             | http://www.canonical.com

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Should we be reverting iptables to iptables-legacy for eoan?

Jamie Strandboge-3
In reply to this post by Stéphane Graber
On Tue, 10 Sep 2019, Stéphane Graber wrote:

> For LXD specifically, we think it would take us about 3 weeks of
> engineering work to sort this in a way that can work on all
> distributions, properly detecting and supporting:
>  - no nft present
>  - nft present but old iptables used
>  - nft present and used

I realize that LXD is an atypical snap, but this is exactly the sort of
thing I would hope that snapd could help with so all snaps wouldn't be
required to go through the same hoops to avoid breaking the system. The
one good thing is that iptables-nft is (supposed to ;) wholly support
the iptables-legacy syntax, so there is some hope of making this easier
for the average snap developer. Those that want full-on netfilter would
need to jump through these hoops of course.

For all systems, it probably makes sense to have a small tool that can
perform this detection, so admins, application developers and the like
can then just focus on what tool to use (ideally upstreamed into
iptables itself).

--
Jamie Strandboge             | http://www.canonical.com

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Should we be reverting iptables to iptables-legacy for eoan?

Stéphane Graber-2
In reply to this post by Jamie Strandboge-3
On Tue, Sep 10, 2019 at 8:12 PM Jamie Strandboge <[hidden email]> wrote:

>
> On Tue, 10 Sep 2019, Julian Andres Klode wrote:
>
> > Hi folks,
> >
> > it turns out that lxd is broken by iptables now using the nft
> > based stuff, because lxd is still using the legacy one from
> > inside the snap.
> >
> > This provides a terrible experience because networking in lxd
> > is not working at all once you enable ufw.
>
> Is this because ufw uses whatever is on the system (ie, nft default) and
> lxd is using whatever is in the snap (ie, historic iptables which with
> iptables 1.8 translates to 'iptables-legacy') and this means that both
> backends are in use, which results in a mixed view of things?

Right, ufw goes through nft, setting up a bunch of rejects, then lxd
starts up, loads its own rules which would normally allow its traffic
but this doesn't work as the nft ruleset runs in its entirety ahead of
running the legacy iptables rules and so a final reject decision was
already made. There are in theory some ways you can have both nft and
legacy working at the same time, though that ends up being a
debugging/management nightmare as there are no tools that query both
and give you a unified view of what's going on.

> The man pages for iptables-nft/iptables-legacy don't call this out as a
> problem, but README.Debian does:
>
>   NOTE: make sure you don't mix iptables-legacy and iptables-nft
>   rulesets in the same machine at the same time just for sanity and to
>   avoid unexpected behaviours in your network.
>
> > I'd suggest we increase the priority of iptables-legacy for eoan,
> > so that it is the default, and move the switch to xtables-nft-based
> > one to next release.
> >
> > This will allow us to have working lxd networking, and gives
> > the lxd team some breathing room.
>
> This makes sense to me. Besides, there are still bugs trickling in where
> the nft backend isn't acting fully compatibly anyway. However, it would
> be nice to use the nft backend by default by 20.04, ie, at the opening
> of the cycle.

Agreed. We've added an item on the LXD roadmap to deal with nft next
cycle. We've had requests coming in from users of other distros
already and I've done a fair amount of research around nft ahead of
this work, but it will require quite a lot of changes on our side as
we are using a fair amount of traditional iptables/ip6tables and then
a bunch of very weird ebtables (which is what failed when run with
compat wrappers in our case).

> That said, this is a very real immediate problem for snaps on 19.10+,
> Buster+ and Ubuntu Core (CC'ing Samuele and Michael specifically) since
> the only available bases are based on xenial and bionic and snaps can
> only stage-packages from xenial and bionic, so snaps that plugs
> 'firewall-control' (or classic snaps) will by definition by far tend to
> use 'legacy' (unless they build iptables from source). Even though the
> system might have the nft backend enabled. Furthermore, the man page for
> iptables-nft says that kernel >= 4.17 is required for the nft backend
> (but ISTR issues in Debian with > 4.19) and its entirely possible
> someone could be running a system with an older kernel with a newer user
> space (eg, core20 snap with bionic kernel snap).
>
> I'm not sure how to fix this. The only thing that seems reasonable is
> for snaps to only be able to use the correct one that matches the
> host/kernel. To pull that off snapd would need to examine the
> capabilities and configuration of the running system then bind mount
> iptables/etc from the host into runtime of the snap (the review-tools
> could flag if snaps ship these binaries). Alternatively, an iptables
> snapcraft part could be created that at runtime can interrogate whether
> to use the nft or legacy backends (or snapd exposes which backend is in
> use to the snap so it can decide).
>
> I suspect that Michael and Samuele will want to move the snap-specific
> parts of this discussion to the snapcraft forum, but +1 from me to
> prefer 'legacy' for the time being in Ubuntu.

I opened https://bugs.launchpad.net/ubuntu/+source/iptables/+bug/1843468
to track this on LP.
If going with the revert, this should get closed and another bug be
opened to track the work to properly switch to nft early next cycle.


Stéphane

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel