How to package Nuxeo DM, a Java EE application, in Ubuntu?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to package Nuxeo DM, a Java EE application, in Ubuntu?

sfermigier
Here are some thoughts and questions about how to package a Java EE application so that it can be accepted in partner.

We have packages that have been created a few months ago, and would be really pleased if they could be included in Natty, so your help would be really appreciated.

  S.

--

# Problem statement

We want to package Nuxeo DM (see: www.nuxeo.org), an open source (LGPL) document management solution developed using Java EE, for Ubuntu (and Debian, and other Linux distributions).

We have created packages 6 months ago, that have been perfected with the help of our user community:


We are fully aware that our packages are not built in a way similar to the way a Linux package is usually built (i.e.: ./configure ; make ; make install). But we believe that:

1. We don't have another reasonable choice for how to build these packages.

2. The issues (and discrepencies with the packaging guidelines for Ubuntu or Debian) are not specific to our project, but common to every project that uses Maven (which seems to be the most popular build tool for Java projects these days) as its main build tool, and more generally to every large-scale Java EE application (such as: XWiki, OpenBravo, Compiere, Open-Xchange, OBM...).

Here are the main objection that have been raisedabout the way we are making our packages:

1. "It looks like they're bundling their own Tomcat.  We haven't allowed
this in the past. Ask that they use our version"

2. "They bundle a TON of JARs, many of which we provide. We may be able to
work with this, but ideally you will want to use our jars where possible."

And our answers:

Point 1.

There are two issues heres:

a. We're not using a "stock" Tomcat distribution, but one "patched" by adding a few jars in the "lib" directory, which means that other applications which would like to use the same tomcat instance could end-up with unexpected behaviour.

b. We're not using any version of Tomcat, but the one that has been proven (by our test suite and manual QA process) to work properly. While it's probable that other versions of Tomcat could also work, we have no proof of it will unless we base our own "standard" distribution on the exact Tomcat version that's shipped with Ubuntu.

Point 2.

It's possible that some of the jars contained in Nuxeo DM are also provided as Ubuntu packages. We could spend more time trying to come up with a list of matches, but I think it is highly improbable that a great many of them would exactly match the version of the jars we provide in Nuxeo.

The problem is, we run our quality assurance process against and have our customers running on an application build from a list of very specific versions of these jars. It is possible that changing a jar version for another won't change anything in the application behavior, but we can't guarantee it, and our experience is that it is not just a theoretical issue, but something we run into every time we upgrade the version of one of our dependencies.

# Actions taken and issues encountered

We could do the following:

Issue 1.: We could make a script that would setup a separate tomcat instance based on the stock Ubuntu Tomcat package, and use it as the basis for the Nuxeo DM package.

BUT: this doesn't solve the fact that we would need to align our own development on the exact Tomcat version you are using, and support different Tomcat version each time you change it for a new Ubuntu release (at the moment, the same Nuxeo DM package can be used on several Ubuntu and Debian releases, as long as Java 6 is installed).

Issue 2.: We could create a .deb package for each of the 250 third-party jars that are contained in Nuxeo DM, at the expense of lot of time and additional complexity, but what would it gain? Each jar wold have to be named with its exact version number, because, and once again this is our experience, we don't expect that changing jar X.Y.Z to X.Y.Z' will never ever end up in some breakage somewhere. So eventually this won't probably gain us anything, because each of the Java EE application you could eventually want to package will have slightly different versions for all the jars that they have validated, and so you will end up with mutiple versions of the jars in a system that has several of these applications installed (or several copies of the jars in the packages repositories).

We understand that this looks counter-intuitive to the Linux / C / C++ developers, but our experience with open source Java development is that you have to be very careful about changing the version of your librairies.     

So, for both of the issues that have been raised, solutions exist that could address the lack of "purity" of our packages, but don't gain anything meaningful for the users (and probably add more complexity), while costing us a great deal of time, effort, and adding additionnal complexity and risk to our own development process (and probably yours too - do you really want to add 250 news packages to the Ubuntu distribution just for 1 application?).

# Asking for a solution to the community if none are found

Maybe you have solutions for the two points that have been raised. Maybe when packaging one of the other Java EE applications that I have mentionned (or some others) someone has found a better way to do it.

But at this point, the *only Java EE package* I see in your partner repository in openbravo-erp (http://archive.canonical.com/ubuntu/pool/partner/o/openbravo-erp/). Looking at the .deb, I seen they are embedding 92 jars:

darkstar% dpkg --contents openbravo-erp_2.50MP-25EU1-1maverick1_all.deb > /tmp/openbravo-erp.contents
darkstar% egrep "\.jar$" /tmp/openbravo-erp.contents| wc -w
   92

This is less than us (more than 250 third-party jars + 187 of our owns), but this is just because their application is less complex than ours, not because they are packaging it differently.

-- 
Stefane Fermigier, Founder and Chairman, Nuxeo
Open Source, Java EE based, Enterprise Content Management (ECM)
http://www.nuxeo.com/ - +33 1 40 33 79 87 - http://twitter.com/sfermigier
Join the Nuxeo Group on LinkedIn: http://linkedin.com/groups?gid=43314
New Nuxeo release: http://nuxeo.com/dm54
"There's no such thing as can't. You always have a choice."


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: How to package Nuxeo DM, a Java EE application, in Ubuntu?

Scott Howard-5
Hello,

On Wed, Feb 2, 2011 at 8:25 AM, Stefane Fermigier <[hidden email]> wrote:
> Here are some thoughts and questions about how to package a Java EE
> application so that it can be accepted in partner.
> We have packages that have been created a few months ago, and would be
> really pleased if they could be included in Natty, so your help would be
> really appreciated.
>   S.

Thanks for wanting to contribute to Ubuntu!

First, if your project is LGPL, you are eligible for Debian Main and
Ubuntu Universe (and possibly Ubuntu main), both of which have more
visibility and easier use than Partner. Usually "Partner" is for
companies that have closed-source code but want to work with Canonical
to distribute their programs. The problem, as you know, are the 200
prepackaged jars - if any of them are not DFSG free, then your package
is contrib/multiverse - and if any of them prevent redistribution,
then we can't touch your project with those jars at all.

The most common path to getting a package into Ubuntu is to get it
into Debian. Ubuntu then automatically syncs your package into Ubuntu.
However, right now Debian's repositories are frozen and their queue
for new packages is quite long, so it probably won't get packaged and
through the queue in time for Natty. You also can upload directly to
Ubuntu's repositories, and link [1] describes the process.

It's probably best if you coordinate with the debian-java team
<[hidden email]>, who will be able to review and upload
your packages. Also see the debian java policy (which also applies to
Ubuntu [2]).

I have some experience with debian-java, so I can comment on your two points:

1) your have to use the ubuntu/debian Tomcat, but a script to set up a
custom instance could be possible as long as it doesn't cause problems
with other parts of the system (other tomcat instances, for example).

2) The main Debian and Ubuntu archives will never allow prepackaged
jars to be distributed for security reasons (we don't know what
actually is in there) and legal reasons (we don't know what license
each part of the code is). This could be a reason why you would want
to distribute your code in the "Partner" repository, but that process
is a little more opaque - someone else might be able to explain that
process. For "traditional" packages, however, you really should
discuss this with the debian-java team. They have volunteers who might
be interested in helping you get some of those packages into Debian
(and thus Ubuntu), although 200 RFP/ITP bugs and packages in the NEW
queue is a bit intimidating - and if anyone one of them has legal
problems (e.g. license incompatibilities) or technical problems (e.g.
not-sane build systems, missing source code), that would hold up the
whole packaging effort. Is there a "distribution" that has the source
code and builds all those third party libraries? We could then package
that distribution as a source package and each of the libraries as a
separate binary packages (also requiring a lot of work, but javahelper
[3] could help in that situation). It is possible to package different
versions of the same java library in Debian/Ubuntu (see [2]), but you
must use a version that is in Debian/Ubuntu.

In conclusion: it appears that it would be a lot of work to get the
package in Ubuntu the traditional way, and you should really discuss
this with the debian-java team. However, the "Partner" way could work
for you - but I don't know how to start that process.

Cheers,
Scott


[1] https://wiki.ubuntu.com/UbuntuDevelopment/NewPackages
[2] http://www.debian.org/doc/packaging-manuals/java-policy/
[3] http://pkg-java.alioth.debian.org/docs/tutorial.html

--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Reply | Threaded
Open this post in threaded view
|

Re: How to package Nuxeo DM, a Java EE application, in Ubuntu?

sfermigier

On Feb 3, 2011, at 4:13 PM, Scott Howard wrote:

> Hello,
>
> On Wed, Feb 2, 2011 at 8:25 AM, Stefane Fermigier <[hidden email]> wrote:
>> Here are some thoughts and questions about how to package a Java EE
>> application so that it can be accepted in partner.
>> We have packages that have been created a few months ago, and would be
>> really pleased if they could be included in Natty, so your help would be
>> really appreciated.
>>   S.
>
> Thanks for wanting to contribute to Ubuntu!
>
> First, if your project is LGPL, you are eligible for Debian Main and
> Ubuntu Universe (and possibly Ubuntu main),

We would be happy to, in the longer term.

> both of which have more
> visibility and easier use than Partner. Usually "Partner" is for
> companies that have closed-source code but want to work with Canonical
> to distribute their programs.

Given that we anticipated some of the objections that have been raised, we initially wanted to target the "partner" repo to get things moving fast. We're not closed-source, but if it can get our packages faster in the hands of Ubuntu users, we would be glad to act as if we were so.

> The problem, as you know, are the 200
> prepackaged jars - if any of them are not DFSG free, then your package
> is contrib/multiverse - and if any of them prevent redistribution,
> then we can't touch your project with those jars at all.

We're an open source project. Of course all our dependencies are open source (and DFSG compliant).

But we don't expect to people taking our word for granted. We can provide a spreadsheet of all the dependencies and their license, so that anyone can check that this is really the case.

> The most common path to getting a package into Ubuntu is to get it
> into Debian. Ubuntu then automatically syncs your package into Ubuntu.
> However, right now Debian's repositories are frozen and their queue
> for new packages is quite long, so it probably won't get packaged and
> through the queue in time for Natty.

Since speed was an issue, we didn't want to got this route, initially.

> You also can upload directly to
> Ubuntu's repositories, and link [1] describes the process.

Whatever is the fastest, although in the long term we would be happier if there were Debian packages for Nuxeo because of the additional reach.

>
> It's probably best if you coordinate with the debian-java team
> <[hidden email]>, who will be able to review and upload
> your packages. Also see the debian java policy (which also applies to
> Ubuntu [2]).

I will.

> I have some experience with debian-java, so I can comment on your two points:
>
> 1) your have to use the ubuntu/debian Tomcat, but a script to set up a
> custom instance could be possible as long as it doesn't cause problems
> with other parts of the system (other tomcat instances, for example).

As I said, this is a problem for us. We can try to work around this issue in the longer term, but this will take additional engineering resources on our side.

> 2) The main Debian and Ubuntu archives will never allow prepackaged
> jars to be distributed for security reasons (we don't know what
> actually is in there) and legal reasons (we don't know what license
> each part of the code is).

That's part of the reason we were targeting "partners", initially.

> This could be a reason why you would want
> to distribute your code in the "Partner" repository, but that process
> is a little more opaque - someone else might be able to explain that
> process.

Please (someone else) do, as all we've been told is to write this email to the ubuntu-devel mailing list and try to work things out.

> For "traditional" packages, however, you really should
> discuss this with the debian-java team. They have volunteers who might
> be interested in helping you get some of those packages into Debian
> (and thus Ubuntu), although 200 RFP/ITP bugs and packages in the NEW
> queue is a bit intimidating

As I said, it's not just about "getting 200 java packages in the distrib", but "getting 200 java package *with the exact version number we know works for Nuxeo in the distrib."

> - and if anyone one of them has legal
> problems (e.g. license incompatibilities) or technical problems (e.g.
> not-sane build systems, missing source code), that would hold up the
> whole packaging effort. Is there a "distribution" that has the source
> code and builds all those third party libraries?

All these jars (except for a few we've had to patch and rebuild ourselves) are from the main public open source jar repository, repo1.maven.org. I don't know how this repository is built, except that it's by the Apache Foundation.

> We could then package
> that distribution as a source package and each of the libraries as a
> separate binary packages (also requiring a lot of work, but javahelper
> [3] could help in that situation). It is possible to package different
> versions of the same java library in Debian/Ubuntu (see [2]), but you
> must use a version that is in Debian/Ubuntu.

As I said, it's a lot of work.

> In conclusion: it appears that it would be a lot of work to get the
> package in Ubuntu the traditional way, and you should really discuss
> this with the debian-java team. However, the "Partner" way could work
> for you - but I don't know how to start that process.

The persons managing the "partner" repo asked me to contact this list, so I feel a bit like a dog chasing his own tail here :(

  S.

--
Stefane Fermigier, Founder and Chairman, Nuxeo
Open Source, Java EE based, Enterprise Content Management (ECM)
http://www.nuxeo.com/ - +33 1 40 33 79 87 - http://twitter.com/sfermigier
Join the Nuxeo Group on LinkedIn: http://linkedin.com/groups?gid=43314
New Nuxeo release: http://nuxeo.com/dm54
"There's no such thing as can't. You always have a choice."


--
ubuntu-devel mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel