Bug 237913 - devel/apache-ant - upgrade, build from source
Summary: devel/apache-ant - upgrade, build from source
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-java (Nobody)
Keywords: patch
Depends on:
Reported: 2019-05-15 16:40 UTC by Mikhail Teterin
Modified: 2020-04-16 13:47 UTC (History)
1 user (show)

See Also:
bugzilla: maintainer-feedback? (java)

Build ant-1.10.6 from source (6.04 KB, patch)
2019-05-15 16:40 UTC, Mikhail Teterin
no flags Details | Diff
Build ant-1.10.6 from source (6.28 KB, patch)
2019-05-16 20:12 UTC, Mikhail Teterin
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Mikhail Teterin freebsd_committer 2019-05-15 16:40:58 UTC
Created attachment 204393 [details]
Build ant-1.10.6 from source

The port currently installs pre-built JARs and documentation, which is a tad depressing -- FreeBSD ports ought to build open-sourced software from source, using binaries only when sources aren't available or in other exceptional circumstances...

The proposed patch corrects this problem -- unfortunately common for the Java-related ports -- and also upgrades the version from 1.10.5 to the currently-latest 1.10.6.

It also enables the test-target.

What's still to do:
 * Some of tests fail, which needs scrupulous investigation;
 * Building of Javadocs throws errors because the optional 3rd-party JARs aren't present -- [bug 63438](https://bz.apache.org/bugzilla/show_bug.cgi?id=63438) filed with the upstram. These errors are currently ignored.
 * The port's options should list all of the optional dependencies -- enabling and disabling the extra features explicitly

These to-do items can be addressed incrementally -- I think, the port can be upgraded as proposed already.
Comment 1 Michael Osipov 2019-05-16 09:32:04 UTC
Hey there, I'd like to make a few comments (specific and in general):

Being an ASF member and happy FreeBSD user, but not an Ant committer have have these cases once in a while. I just can tell how we handle thia at least with Maven, but am sure that other projects do similarly:

ASF only releases source code, in some cases like Java, we do provide binary releases. Both are trustworthly because all of them signed (GPG) when you download from dist.apache.org. In general we reject custom-built binary distributions becaue they aren't verified by us and contain sometimes tremendous modifications, .e.g., Debian and friends are worstcase here, I just loath their approach. We refuse to evaluate bugs with those customs distros. Of course, it requires some adaptations for a specific platform, but it has to be reasonable and generally not break the flow of the application.

As for Javadoc issues: you have have the ports system download mutltiple files, so you could download them from Maven Central and add them for the build OR simply disable the build of Javadoc because I do not see any benefit generating it here if you use Ant from command line only. They are needed by people wrigin custom targets/commands for Ant and they will use an IDE where source JARs are pulled from Central too.
Comment 2 Mikhail Teterin freebsd_committer 2019-05-16 10:53:14 UTC
(In reply to Michael Osipov from comment #1)
You seem to agree, that building from source is preferable.

As for Javadoc, whether to build it is controlled by the user-settable option. The DOCS-option is standard, and can be set globally -- for all ports -- or individually When it is set, however, it must work....

The current version of the port respects it already -- just implements it differently, by downloading the prebuilt documentation, instead of regenerating it. Which sends us back to it being preferable to build things from source.
Comment 3 Michael Osipov 2019-05-16 11:15:41 UTC
(In reply to Mikhail Teterin from comment #2)

Being able to reproduce open source softare is a must. If I can't build it my own ony my platforms, I won't use it. It's that simple. It does not necessary mean that I do not trust others to do this for me.

Af for the DOCS, I really would recommend get the JARs as intended by the build. Try to deviate as little as possible from the official binary packages.
Comment 4 Mikhail Teterin freebsd_committer 2019-05-16 20:12:26 UTC
Created attachment 204405 [details]
Build ant-1.10.6 from source

The earlier version was not running javadocs, but still installing other ant manuals (in HTML). This update disables those as well by not even extracting them.
Comment 5 Greg Lewis freebsd_committer 2019-06-27 18:13:28 UTC
I would recommend against building ant from source.

Aside from the issues Michael mentions, which are significant, we've seen in the past that this created complexities in the port.  This includes:

* Building with a certain JDK version means that older JDKs can't run the classes that are built.  E.g. if I build with openjdk11 then openjdk8 won't be able to execute ant.  If you force the build to use the oldest JDK available in ports then you force users who may not want it to have to install it.  In general the packaged JARs are built with the oldest JDK that the source supports and this can then be executed by any newer JDK, without the user having to install the older JDK.

* Some Java based projects provide different functionality based on the version of the JDK they are built with.  I don't think ant falls into that category, but in general once we start building Java ports from source we run into problems where we're not sure what functionality we have since it depends on the version of the JDK it was built with.  This becomes a mess.

Overall, the Java community tends to distribute and use compiled class code as standard.  We've tried going down the "build from source" route before, but it turned out to add a lot of complexity and maintenance without providing any real benefits.  Particularly for a vendor like ASF I think we should be comfortable using their binary packages.
Comment 6 Michael Osipov 2019-06-27 18:51:13 UTC
(In reply to Greg Lewis from comment #5)

I second that. E.g., I am the portmaintainer for Sonatype Nexus OSS, building this from source is as timeconsuming as building LLVM or GCC from source. It does not add any benefit. All ASF releases are done by voting and testing, and since everything is signed you can always contact the release manager @apache.org and ask for help. I'd am always open to request for all Maven modules I release.
Comment 7 Mikhail Teterin freebsd_committer 2019-06-27 20:54:44 UTC
Greg, Michael, the arguments you raise would apply - verbatim - to the entire FreeBSD. Indeed, building from source "adds complexity". It also targets a certain hardware. And so on.

Yet, FreeBSD is an open source project -- with the sources provided for EVERYTHING, with it being possible to rebuild every file in the distribution (including documentation)...

And ports are -- or should be -- the same. Quoting from https://www.freebsd.org/ports/ (emphasis mine):

  Each "port" listed here contains any patches necessary to make the
  original application SOURCE CODE compile and run on FreeBSD.

It is quite disappointing to see a number of Java ports violate the above principle installing precompiled binaries -- simply repackaging them from some format into another (pkg a.k.a tar.xz). Of course, it is understood, that in a volunteer project developers may simply not have time for a proper implementation.

Yet, when the work's already been done and requires only minor fixups (if any!), to reject it on the grounds, that "building from source is not necessary", is "too complicated", or "takes too long", is contrary to the very purpose of the whole endeavor.

Indeed, in Comment #3 Michael seems to agree with that -- only to contradict himself later...

That's in general. As for particulars:

> Building with a certain JDK version means that older JDKs can't run the classes that are built.

This is not quite true. The "-target" argument can be added to javac's command line to target an older JVM, while using a newer one. This is similar to using C-compiler's -march flag...

Indeed, by forcing the users to accept the binaries built for the lowest version (JDK7?), you're pessimizing the situation for the users of newer JDKs...

> If you force the build to use the oldest JDK

The proposed patch would use whatever JDK the user *already has* installed. The result will be perfectly usable for almost all users...

> building this from source is as timeconsuming as building LLVM

"buildworld" once took my 486-33 computer 4 days. Was it a mistake for FreeBSD to bother maintaining the src/ tree buildable from source back then?

Whoever has no time to build from source, can always install a prebuilt RPM, oops, package. But a) it should be possible to build from source; and b) such building should be regularly exercised to prevent bit-rot.

This is not a problem unique to Java ports and needs no Java-specific solutions.
Comment 8 Michael Osipov 2019-06-27 21:05:22 UTC
The last statement isn't meant in general. I love to build from source, in fact, if I can't build OSS from source, I won't use it.

-- Therefore, I am grateful for the work you all did. Hopefully the Ant port maintainer can have a look at it.

But would you also build all transitive JARs of a Java project form source? And add each one as a port?
Comment 9 Mikhail Teterin freebsd_committer 2019-06-28 13:14:43 UTC
(In reply to Michael Osipov from comment #8)
> But would you also build all transitive JARs of a Java project form source?

Do you mean the commonly-bundled JARs like log4j or junit? Yes, absolutely -- ports should depend on each one's respective port instead of the code bundled with the source, whether the bundle is a binary JAR or the Java-source.

> And add each one as a port?

Yes, following the principles and for the reasons put forth in:

Comment 10 Michael Osipov 2019-06-28 21:21:15 UTC
(In reply to Mikhail Teterin from comment #9)

This may and will contradict the stable approach with fixed versioned in the POM. Ultimately you may have situations where teo ports require one dep in two versions. How to solve?
Comment 11 Greg Lewis freebsd_committer 2019-06-28 23:19:02 UTC
Mikhail, my biggest concern with what you're proposing is that this is how the Java ports used to be constructed at one point, and it didn't work out well at all.  I was once a believer in them building from source too, but the practicality is that many of the Java packages you might want to have as ports use Maven as the build system, which allows them to specify exact versions of other packages to depend on, which are sourced from a network server during the build.  This doesn't work for the ports system and messing around with patches for all of that build infrastructure and managing all those dependencies somehow is a massive amount of work.  Not to mention the other problems such as projects ignoring problem reports from FreeBSD users since we aren't using the official releases from them.

The first step in shifting over to building from source is not to insist on changing ports back to building from source.  Rather the first step is to propose practical solutions for the problems that lead us away from that previously.  To do otherwise is just starting down a path that leads to failure.
Comment 12 Mikhail Teterin freebsd_committer 2019-06-29 00:28:46 UTC
(In reply to Greg Lewis from comment #11)
> use Maven as the build system

Let's start small, maybe? With the software, that does NOT build using Maven?

> which allows them to specify exact versions of other packages to depend on

OpenOffice.org used to (still does) bundle all sorts of things with it too -- libxml, DB-clients, libjpeg, png, etc. But the port, gradually, evolved to make it use the ports-provided alternatives.

It can be done.
Comment 13 Michael Osipov 2020-03-07 15:46:30 UTC
>  Not to mention the other problems such as projects ignoring problem reports from FreeBSD users since we aren't using the official releases from them.

Not just FreeBSD users, most ASF project reject to support custom build for various reasons. Debian is worst because the rip everything to pointless pieces.
Comment 14 Mikhail Teterin freebsd_committer 2020-03-11 03:29:20 UTC
Are we still debating, whether a port of 3rd party software to FreeBSD (an operating system one can rebuild from source entirely) should be installed from binaries vs. compiled from source?

Comment 15 Michael Osipov 2020-03-11 08:22:45 UTC
(In reply to Mikhail Teterin from comment #14)

I am not trying to convince you for either. If your patch is complete, I can test if you want.