netplan and post-up/pre-down scripts

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

netplan and post-up/pre-down scripts

Mike Pontillo-2
Hi all,

   Recently, I was working on a project that led me to become frustrated with the current state of `systemd` and `ifupdown` (e.g. /etc/network/interfaces or /e/n/i) in Xenial. I remembered that `netplan`[1] was under development, so I added the PPA for Xenial and gave it a try. Unfortunately, it didn't solve the issue for me[2]... but I also realized there was a blocking issue in the way of my adoption of it.

   Let me explain my use case: when an interface goes up or down, I want to be able to do event-driven things with the network configuration, such as add or remove routes, run a DHCP client, etc. My first attempt to make this happen was to add `post-up` and `pre-down` scripts to do this. However, this had a fatal flaw for my application: `ifupdown` doesn't separate the concept of operational status from the concept of administrative status. (That is, in `ifupdown`, an interface is "up" if the admin says it is up. Link up or link down does not seem to matter; it's strictly an /administrative/ status[3].)

   Long story short: in order to get the behavior I wanted, I wrote a custom script that monitors *operational status* (aka physical link up/down status), and I launch it using /e/n/i's `post-up`, and bring it down using /e/n/i's `pre-down` scripts.

   Looking at the `netplan` spec[4], I don't see a way to achieve that functionality. I know that many people are using the script-callout functionality in /e/n/i to achieve what they need to achieve, so it seems to me that having this in `netplan` is critical to achieving parity with what we have in Xenial with `ifupdown`.

   In an ideal world, I think `netplan` would be able to model my use case out-of-the-box.[5] But since we can't expect to model everyone's use case, it seems like custom scripting functionality is a hard requirement, though perhaps one that could have tricky cross-platform implications.

   Thoughts?

Regards,
Mike


[2]: The behavior was exactly the same, which is actually good for consistency.

[3]: Well, it is after an annoying 5 minute timeout, in some cases.

[4]: https://git.launchpad.net/netplan/tree/doc/netplan.md

[5]: As an aside, this brings up an interesting subtle difference between rendering a networking configuration with (networkd, ifupdown) vs NetworkManager. In NetworkManager, there *IS* a concept of operational status, and I would have been able to get closer to the behavior I wanted, if I was willing to pull in all those dependencies. We should be mindful that rendering the same configuration on NetworkManager, even for simple things like a static IP address and a DHCP interface, could produce drastically different behavior than with networkd or ifupdown.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: netplan and post-up/pre-down scripts

Martin Pitt-4
Hello Mike,

Mike Pontillo [2017-01-06 10:12 -0800]:
>    Recently, I was working on a project that led me to become frustrated
> with the current state of `systemd` and `ifupdown` (e.g.
> /etc/network/interfaces or /e/n/i) in Xenial. I remembered that
> `netplan`[1] was under development, so I added the PPA for Xenial

Note that netplan is in xenial-updates, so no PPA needed.

>    Let me explain my use case: when an interface goes up or down, I want to
> be able to do event-driven things with the network configuration, such as
> add or remove routes, run a DHCP client, etc.

These two and more are already supported by networkd and NM (i. e. both of
netplan's current backends) and also in the netplan YAML itself of course. OOI,
what is your particular use case?

> My first attempt to make this happen was to add `post-up` and `pre-down`
> scripts to do this. However, this had a fatal flaw for my application:
> `ifupdown` doesn't separate the concept of operational status from the
> concept of administrative status.  (That is, in `ifupdown`, an interface is
> "up" if the admin says it is up.  Link up or link down does not seem to
> matter; it's strictly an /administrative/ status[3].)

The ifup@.service (more or less) deals with hotplugging, so normally as an
admin you would not explicitly "ifup" any interface (unless you mark them as
"manual", but then you are on your own anyway). So I fail to see the problem
here -- for "auto" and "allow-hotplug" interfaces this should just work?

> Looking at the `netplan` spec[4], I don't see a way to achieve that
> functionality

Correct. NetworkManager calls /etc/network/if-up.d/ when an interface is up
(same as ifupdown), but networkd doesn't, and it's not planned upstream to do
that. It could be done by monitoring networkd's D-Bus signals and reacting to
it, but so far there hasn't been a pressing use case for this. Many existing
if-up.d/ scripts are workarounds for software which aren't written with the
idea of network hotplugging in mind (e. g. not using IP_FREEBIND), and many
others aren't necessary or even actively detrimental with NM/networkd as they
are essentially ifupdown implementation of bridges or similar, so they must not
be called with NM/networkd.

> I know that many people are using the script-callout functionality in /e/n/i
> to achieve what they need to achieve

Actually, /etc/network/if-up.d/ has been the much more popular approach AFAIK.

> In an ideal world, I think `netplan` would be able to model my use case
> out-of-the-box.[5] But since we can't expect to model everyone's use case,
> it seems like custom scripting functionality is a hard requirement, though
> perhaps one that could have tricky cross-platform implications.

If you need this, then I suggest to use the NM backend, which gives you
/etc/network/if-up.d/. We will never use NM in confined scenarios like the
initramfs, so that should be reasonably safe. OTOH netplan itself (with
networkd) was meant to work in initrd and other early-boot scenarios where
arbitrary script callouts are not supportable.

If you have something particular that needs to be set/done when an interface
goes up, I suggest filing a bug -- maybe that functionality even already exists
in networkd/NM and just needs to be wired up to YAML?

Thanks,

Martin

--
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: netplan and post-up/pre-down scripts

Mike Pontillo-2
Hi Martin,

Thanks for the reply.

>    Let me explain my use case: when an interface goes up or down, I want to
> be able to do event-driven things with the network configuration, such as
> add or remove routes, run a DHCP client, etc.

These two and more are already supported by networkd and NM (i. e. both of
netplan's current backends) and also in the netplan YAML itself of course. OOI,
what is your particular use case?

I had two test cases, which I first tried implementing using 'auto' and 'dhcp' interfaces in ifupdown, then in netplan.

Physical setup: two interfaces, intended to be used for routing. One to be used for a WAN interface, the other to be used for a LAN interface.

Logical setup: each physical interface has an attached bridge, each containing just that one port. For example:

Physical: eth0 ---> Bridge: wan0
Physical: eth1 ---> Bridge: lan0

I configured it this way because I wanted a no-hassle way to attach LXD containers and/or VMs to a physical network.[1]

Now, this may have a lot to do with the fact that I'm using the bridge; the behavior I'm describing may be very different for a "pure physical" interface. But with this setup, I see 5-minute timeouts at boot when I:

 - Declare an 'auto' interface in 'ifupdown' which is disconnected at system boot.
 - Declare a 'dhcp' interface in 'ifupdown' which is disconnected at system boot.

I've seen similar 5-minute timeouts occur in the field with bond interfaces whose physical links are not [all?] available.

After the system booted, I then tried more tests, such as checking if routes and addresses were added or removed when I disconnected the physical link. I was also testing a redundant setup with interfaces on the same network, and a route metric in place for shortest-path selection. So I expected the higher-metric route to be used when the lower-metric interface's link went down. However, the behavior I saw was that the routes via the downed interface were marked "linkdown" in the "ip route" output, but route lookups still selected the "linkdown" route rather than switching to the higher-metric route to the same destination. (So the only option is to remove them from the kernel upon link down. Note, I don't want to configure a bond; this is for "someone accidentally, or maybe on purpose, plugged the WAN port into the LAN, or bridged the networks"; you want traffic to keep flowing in that case, and return to normal when it's fixed.)

I would have to go back and test more to confirm, but I suspect some of these issues have to do with the bridge. The behavior I want is for the *bridge* to, for example, release its DHCP address, or delete its routes, if the *underlying* physical interface goes down, and bring up the DHCP client and/or static routes if the link comes back up. That way, any running containers correctly see "no route to host" ICMP messages, rather than trying to route packets to an interface whose underlying link is down.

I ended up having to write the configuration in 'ifudown' with "auto" and "manual" on all the interfaces, and write a separate script (called from post-up and pre-down callouts) to monitor link status, so that I could take action on the corresponding bridge when the underlying link goes up or down.

 
> My first attempt to make this happen was to add `post-up` and `pre-down`
> scripts to do this. However, this had a fatal flaw for my application:
> `ifupdown` doesn't separate the concept of operational status from the
> concept of administrative status.  (That is, in `ifupdown`, an interface is
> "up" if the admin says it is up.  Link up or link down does not seem to
> matter; it's strictly an /administrative/ status[3].)

The ifup@.service (more or less) deals with hotplugging, so normally as an
admin you would not explicitly "ifup" any interface (unless you mark them as
"manual", but then you are on your own anyway). So I fail to see the problem
here -- for "auto" and "allow-hotplug" interfaces this should just work?

I have not used allow-hotplug, to be honest. The documentation isn't clear on how exactly it works; I'm not sure what exactly triggers the hotplug event. This might solve part of the issue, in that I want to allow booting the system even if the link is down. (Is this the officially-supported way [across all backends] to get rid of the 5-minute timeout I love to hate?) But I also want to be able to take action on link down. So I'll have to look into it more.[2]

If you need this, then I suggest to use the NM backend, which gives you
/etc/network/if-up.d/. We will never use NM in confined scenarios like the
initramfs, so that should be reasonably safe. OTOH netplan itself (with
networkd) was meant to work in initrd and other early-boot scenarios where
arbitrary script callouts are not supportable.

Again, this is for a router, so NetworkManager doesn't seem like a good option. For the record, I'm fine with the early-boot limitation. I just want the flexibility to both cause and solve my own problems. ;-)

But at this point, maybe I'm using so little of 'ifupdown' that I might as well not configure anything, and do everything in custom scripts. But I'd hate for everyone in my situation to have to do that.

Regards,
Mike

[1]:  You could also throw in some VLANs, to add to the complexity. I have at least one, but it might not be relevant to this discussion. (I think I had a eth0.100 and was planning to create a bridge on top of that as well, to attach virtual interfaces without giving them access to every VLAN on the physical link.) This probably causes similar issues if set to 'auto'; I wonder if 'allow-hotplug' will work.

[2]: So far, I think "allow-hotplug" is a little strange in that seems to blur the distinction between operational status and administrative status. (I guess you could say that the administrative status is "always up" if you have "allow-hotplug" in your /etc/network/interfaces. But what if a flaky NIC is toggling on and off rapidly, and you keep getting hotplug events? I suspect 'ifdown' would not be enough to bring the interface down permanently; you'd need to comment out the "allow-hotplug" in /e/n/i first.) Whereas, with my solution, if an interface set to "auto" and "manual" you can just "ifdown" the interface and it would go away until both: (1) "ifup <interface>" occurs, and (2) the interface link is up.


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: netplan and post-up/pre-down scripts

Seth Arnold
On Tue, Jan 10, 2017 at 01:48:03AM -0800, Mike Pontillo wrote:
> Now, this may have a lot to do with the fact that I'm using the bridge; the
> behavior I'm describing may be very different for a "pure physical"
> interface. But with this setup, I see 5-minute timeouts at boot when I:

I'm sorry to say that I did not read this message with the attention it
deserves, but it's been a trying couple of days. So..

Is this a manifestation of the bridge trying to do spanning tree protocol?
Most of the walkthroughs I've read lately about bridged networking include
commands to turn off STP before they start adding ports to the bridge.

If you don't need it, try turning stp off:
brctl stp wan0 off
brctl stp lan0 off

(Is there a way to turn it off via iproute2 instead? I couldn't find it
quickly, if there is a way.)

Thanks

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: netplan and post-up/pre-down scripts

Mark Shuttleworth-3
In reply to this post by Mike Pontillo-2
On 06/01/17 13:12, Mike Pontillo wrote:

>    Long story short: in order to get the behavior I wanted, I wrote a
> custom script that monitors *operational status* (aka physical link
> up/down status), and I launch it using /e/n/i's `post-up`, and bring
> it down using /e/n/i's `pre-down` scripts.
>
>    Looking at the `netplan` spec[4], I don't see a way to achieve that
> functionality. I know that many people are using the script-callout
> functionality in /e/n/i to achieve what they need to achieve, so it
> seems to me that having this in `netplan` is critical to achieving
> parity with what we have in Xenial with `ifupdown`.
>
>    In an ideal world, I think `netplan` would be able to model my use
> case out-of-the-box.[5] But since we can't expect to model everyone's
> use case, it seems like custom scripting functionality is a hard
> requirement, though perhaps one that could have tricky cross-platform
> implications.

Would 'got-link' and 'lost-link' be good names for this?

Mark

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: netplan and post-up/pre-down scripts

Mike Pontillo-2
On Mon, Jan 16, 2017 at 7:35 AM, Mark Shuttleworth <[hidden email]> wrote:
Would 'got-link' and 'lost-link' be good names for this?

I'm not certain a new event name is needed for this functionality; it seems to me that the current definition of 'up' isn't quite correct.[1] (But all this might be a moot point depending on what is supported in networkd, and how it behaves.)

I understand there have been several attempts to address this in the past, such as the 'allow-hotplug' option, ifplugd, ifupdown-extra, NetworkManager, and now networkd. IMHO, no solution is complete unless it properly separates adminStatus from operStatus, and holds off on confirming "link up" until both are "up". For backward compatibility, a boolean flag (similar to "allow-hotplug") should indicate whether or not the system is allowed to continue booting if the interface is down.[2]

Another subtle detail is that if an interface is administratively down, there should be an option to cause the NIC to take its physical link down. That way, whatever is on the other side of the link doesn't assume its peer is active. (This is standard behavior on a router or a switch, but may be atypical for a server... so I think the default behavior should continue be "leave the physical link up".)

Regards,
Mike


[1]: I would refer to the IF-MIB definitions for administrative and operational status, which haven't changed in a long time. They can be found in RFC 2863 sections 3.1.12 and 3.1.13[1]. There is also discussion (amendments to a previous RFC) about when to send the "linkUp" trap. (To summarize, only when a link is both operationally and administratively up.) See the relevant states here:


Contrast this to the default behavior of an auto/static interface: an interface is considered UP if its operStatus AND adminStatus were "up" within 5 minutes of boot. After that, you can throw all your assumptions out the window; the interface will stay DOWN even if its operational status changes from "down" to "up", and the system will hobble along in a half-configured state, even if the link status changes.


[2]: I think that should default to to allow boot, to prevent the UX nightmares that occur during boot when the boot process waits for interfaces it thinks should be up. If a particular service is finicky enough to not handle a missing interface gracefully, the admin can manually configure the flag to /not/ allow boot.

The current behavior is also strange because if an interface becomes operationally "down" after the five minute timeout, the system takes no action, pretending nothing happened. (Why did we just wait 5 minutes for an interface to be up, if we weren't going to care if it later went down?) If a service /seriously/ depends on an interface being up, and cannot handle changes in interface status, the admin should configure that service to start upon receiving a link up event, and stop it upon receiving a link down event for that interface.

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: netplan and post-up/pre-down scripts

Tom H-4
In reply to this post by Seth Arnold
On Thu, Jan 12, 2017 at 10:57 PM, Seth Arnold <[hidden email]> wrote:
>
> If you don't need it, try turning stp off:
> brctl stp wan0 off
> brctl stp lan0 off
>
> (Is there a way to turn it off via iproute2 instead? I couldn't find
> it quickly, if there is a way.)

I've never found any ip or bridge option to show or control stp.
There's only echoing into "/sys/":

# brctl show lxcbr0
bridge name bridge id STP enabled interfaces
lxcbr0 8000.f04da29a3351 yes enp19s0
tap0

# echo 0 > /sys/class/net/lxcbr0/bridge/stp_state

# brctl show lxcbr0
bridge name bridge id STP enabled interfaces
lxcbr0 8000.f04da29a3351 no enp19s0
tap0

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: netplan and post-up/pre-down scripts

Mathieu Trudel-Lapierre-2
In reply to this post by Mike Pontillo-2
On Tue, Jan 17, 2017 at 1:02 AM, Mike Pontillo <[hidden email]> wrote:
On Mon, Jan 16, 2017 at 7:35 AM, Mark Shuttleworth <[hidden email]> wrote:
Would 'got-link' and 'lost-link' be good names for this?

I'm not certain a new event name is needed for this functionality; it seems to me that the current definition of 'up' isn't quite correct.[1] (But all this might be a moot point depending on what is supported in networkd, and how it behaves.)

I understand there have been several attempts to address this in the past, such as the 'allow-hotplug' option, ifplugd, ifupdown-extra, NetworkManager, and now networkd. IMHO, no solution is complete unless it properly separates adminStatus from operStatus, and holds off on confirming "link up" until both are "up". For backward compatibility, a boolean flag (similar to "allow-hotplug") should indicate whether or not the system is allowed to continue booting if the interface is down.[2]

I agree booting should continue for any device that is down even if it's configured and marked as fine to still be down. I think rather than trying to skip a long delay that would happen in some other cases, we should revisit whether it makes sense for it to take 5 minutes; or even make this configurable. 5 minutes is a lot; 1 minute is acceptable in many configurations, 30 seconds is ideal in some. All this only makes sense for DHCP-configured interfaces where the link is up but no DHCP response has been received. If the link is down, there is no point in ever waiting. This may need fixed in networkd and NM to allow configuring the DHCP timeout.

I do not know that any of the different network management tools were planned to address administrative status of an interface. It may even in fact require kernel work (I haven't looked yet) to allow its state to be set to administratively down.

Please file a bug about this. I'll review and look how we can specify this in netplan, and how we can drive it in the various backends... But I expect it will require work in both networkd and NetworkManager before it does work correctly. Servers aren't typically used in such a way, and setting that option seems like it might be quite intrusive (I mean, of course the default wouldn't be for interfaces to be administratively down, but I can see issues coming up from it. One of them is how to do it in the first place, and another is that it would only happen once the renderer (networkd/NM) is up and running, so potentially you've already sent/received packets on the interfaces... ideally, that shouldn't happen, but maybe I'm overthinking it).
 

Another subtle detail is that if an interface is administratively down, there should be an option to cause the NIC to take its physical link down. That way, whatever is on the other side of the link doesn't assume its peer is active. (This is standard behavior on a router or a switch, but may be atypical for a server... so I think the default behavior should continue be "leave the physical link up".)

Of course. If something is administratively down, we must think of it in terms of *powered down*. Otherwise it's simply not configured.

/ Matt

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: netplan and post-up/pre-down scripts

Mike Pontillo-2
In reply to this post by Seth Arnold
Hi Seth,

Sorry for the delayed responses on this thread (this is a side project of mine).

On Thu, Jan 12, 2017 at 7:57 PM, Seth Arnold <[hidden email]> wrote:
Is this a manifestation of the bridge trying to do spanning tree protocol?
Most of the walkthroughs I've read lately about bridged networking include
commands to turn off STP before they start adding ports to the bridge.

My present workaround is a /etc/network/interfaces file whose bridge looks something like this:

iface wan0 inet manual
    metric 5000
    bridge_ports enp1s0
    bridge_stp off
    bridge_waitport 0
    bridge_fd 0
    post-up run-one /root/ifmonitor enp1s0 &
    pre-down kill $(cat /run/ifmonitor.enp1s0) 2> /dev/null || true

So no, this particular issue isn't related to the spanning-tree configuration. (That part was unchanged when I was seeing the 5-minute timeouts.)

The 5-minute timeout was an issue with the interface set to 'auto' and the DHCP configuration present in /e/n/i. I had to replace that with the post-up script which manually monitored the interface and launched (or killed) the DHCP client to get the behavior I wanted (no 5-minute timeout of the WAN interface was disconnected or didn't respond to DHCP).

Regards,
Mike

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: netplan and post-up/pre-down scripts

Mike Pontillo-2
In reply to this post by Mathieu Trudel-Lapierre-2
On Mon, Feb 13, 2017 at 8:21 AM, Mathieu Trudel-Lapierre <[hidden email]> wrote:
I agree booting should continue for any device that is down even if it's configured and marked as fine to still be down. I think rather than trying to skip a long delay that would happen in some other cases, we should revisit whether it makes sense for it to take 5 minutes; or even make this configurable. 5 minutes is a lot; 1 minute is acceptable in many configurations, 30 seconds is ideal in some. All this only makes sense for DHCP-configured interfaces where the link is up but no DHCP response has been received. If the link is down, there is no point in ever waiting. This may need fixed in networkd and NM to allow configuring the DHCP timeout. 

Yes; perhaps even a kernel parameter would be good. That way it could be customized depending on the purpose of the deployment.

For example, I might want a 1-minute wait time before declaring interfaces "up" if I'm deploying a high availability web proxy with multiple bonded interfaces. Or I might want it to not wait at all if I'm deploying a network infrastructure device, where I would expect that the interface configuration needs to be flexible enough to never prevent me from booting the system.

(Though in general I still think we should move toward eliminating the timeout altogether as a long-term goal, though I realize that may be somewhat idealistic.)

I do not know that any of the different network management tools were planned to address administrative status of an interface. It may even in fact require kernel work (I haven't looked yet) to allow its state to be set to administratively down.

In my (very limited) testing, I can do something like "ip link set dev <device> <down|up>" and when I look at "ethtool <device>" I see "Link detected: <yes|no>". (But I'm not sure what the initial state of the interface is before ifupdown touches it.)
 
Please file a bug about this. I'll review and look how we can specify this in netplan, and how we can drive it in the various backends... But I expect it will require work in both networkd and NetworkManager before it does work correctly. Servers aren't typically used in such a way, and setting that option seems like it might be quite intrusive (I mean, of course the default wouldn't be for interfaces to be administratively down, but I can see issues coming up from it. One of them is how to do it in the first place, and another is that it would only happen once the renderer (networkd/NM) is up and running, so potentially you've already sent/received packets on the interfaces... ideally, that shouldn't happen, but maybe I'm overthinking it).

Right; it might make a difference what the initial state of the interface is (before ifupdown, networkd, or NetworkManager take control of it). I'm pretty sure I've observed ifupdown bringing interfaces online, and leaving interfaces offline if they aren't mentioned in /etc/network/interfaces (but I'd have to double check the initial state of the interface). So it might be that this is already happening, at least to some extent.

But I agree; ideally the service which renders the network configuration will bring the interfaces online, to avoid the link coming up before it's ready. (But if we were to iterate on this, I would say that the interface coming up in a 'default up' state for a short time would be a small price to pay on the path toward getting this right.)

I'll sleep on it and try to get a bug filed tomorrow.

Of course. If something is administratively down, we must think of it in terms of *powered down*. Otherwise it's simply not configured. 

Indeed, from the user perspective, it should seem to be powered down (in that if an Ethernet cable is connected which would otherwise show a link light, you would see nothing appear to happen). I think setting the interface link down should be enough to accomplish that; hopefully the driver would know that it could be placed into a power save mode or similar.

Regards,
Mike



--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel