|Summary:||[exp-run] graphics/gdal: Too many necessary options disabled by default|
|Product:||Ports & Packages||Reporter:||Loïc Bartoletti <lbartoletti>|
|Component:||Individual Port(s)||Assignee:||Port Management Team <portmgr>|
|Severity:||Affects Some People||CC:||chitty_cloud, lbartoletti, nezos, pi, rhurlin, sunpoet|
Description Loïc Bartoletti 2017-11-30 21:49:53 UTC
Hi, For my personal case, I think that gdal have to enable by default (for the pkg) some necessary options: Very strong: SQLITE: used for geopackage new lingua franca formats for GIS (by default in QGis 3) CURL: for all web services EXPAT: for spreadsheet (excel / openoffice) and openstreetmap FREEXL: old excel spreadsheet GEOS: used by other and perform some operations Strong: PGSQL : most GIS tools install postgresql Soft: ECW How do you feel about that? Regards. Loïc
Comment 1 Sunpoet Po-Chuan Hsieh 2017-12-09 17:09:13 UTC
There are many different ways to use this port and each of them requires different dependencies. I prefer to keep a minimal dependency for the default option and add more options for people who have different needs to build this port themselves. I guess this problem will be solved when subpackage becomes available.
Comment 2 Loïc Bartoletti 2018-01-16 13:36:33 UTC
Hi, I can understand your point of view, but the problem is that it causes problems for the use of GIS clients (QGis, Grass and maybe SAGA) that are all based on this library. I don't think it's too big to install these dependencies (especially those marked as very strong) Regards.
Comment 3 Loïc Bartoletti 2018-01-17 18:18:58 UTC
With flavors, can we imagine to have little (like today), mid (my suggestion), full (all options) flavors?
Comment 4 Rainer Hurling 2018-01-17 18:41:52 UTC
(In reply to lbartoletti from comment #2) Speaking as the maintainer of graphics/qgis and math/saga I would like to vote for a as much as possible and useful enabled set of (strong) options. Option PGSQL is very important also. Applications like QGIS and SAGA do realize much of there functionality via helper programs, primarily GDAL functions. Thanks and best wishes, Rainer
Comment 5 Loïc Bartoletti 2018-05-23 06:47:24 UTC
Any news? I just checked other systems and the proposed dependencies are similar. Except qgis, grass and saga, this is all GIS tools that need more default options in gdal. Is there any chance we can review this for gdal 2.3.0?
Comment 6 Loïc Bartoletti 2018-05-23 06:52:04 UTC
Also, if you're afraid about dependencies, you can see that some of this are already required by other installed GIS software, just use ports-mgmt/portgraph or ports-mgmt/py-skog to see it. Thank you
Comment 7 Loïc Bartoletti 2019-05-08 12:57:04 UTC
I'm back :) Since QGIS 3, geopackage is the default format. I patched upstream qgis to avoid some error and crash since sqlite is not enabled in our default GDAL. QGIS require also MDAL (internal, but I plan to create a port of it) which require for some case that GDAL have NetCDF/HDF5 enabled too. And now, math/pdal 1.9.0 require that GEOS is enabled in GDAL. I will prepare a patch with this options.
Comment 8 Loïc Bartoletti 2019-05-08 19:04:56 UTC
Created attachment 204276 [details] gdal-2.4.1_2 add flavors for gdal Proposal patch with flavors (including patch from #237298)
Comment 9 Loïc Bartoletti 2019-05-08 19:08:46 UTC
My proposal: default: astro/libosmium databases/grass7 databases/postgis20 databases/postgis22 databases/postgis23 databases/postgis24 databases/postgis25 graphics/R-cran-rgdal graphics/gmt graphics/gmt5 graphics/mapcache graphics/mapnik graphics/mapserver graphics/osgearth graphics/p5-Geo-GDAL graphics/py-fiona graphics/py-gdal graphics/qgis graphics/qgis-ltr math/R-cran-sf math/pdal math/saga lite: astro/merkaartor astro/qmapshack cad/sumo graphics/openorienteering-mapper graphics/cloudcompare games/flightgear-terragear
Comment 10 kunda 2019-06-21 14:20:39 UTC
Just wondering, should I open another ticket to create a separate port for gdal 3.0? https://www.osgeo.org/foundation-news/gdal-3-0-0-is-released/
Comment 11 Loïc Bartoletti 2020-02-12 18:18:04 UTC
Created attachment 211597 [details] GDAL 3 with flavors. Here's the update. It is becoming more and more critical for some applications, including qgis, saga and postgis.
Comment 12 Loïc Bartoletti 2020-02-12 18:19:27 UTC
I hope I've updated all ports. Please let me know.
Comment 13 Loïc Bartoletti 2020-02-12 18:20:47 UTC
Dmitry amdmi3@ what's your preference for osg? Thanks
Comment 14 Dmitry Marakasov 2020-02-12 21:16:01 UTC
> GDAL 3 with flavors. I don't think you may use flavors here as these conflict with each other (e.g. install same files) and different packages depending on different gdal flavors will lead to inability to install such packages together. IMO the key problem is here: > I prefer to keep a minimal dependency for the default option and add more options for people who have different needs to build this port themselves. As long as we want FreeBSD to be useful general purpose system, we must not require users to build anything from ports, as this requires extra time, CPU, diskspace, leads to downtime on upgrade and possible build problems, so we want binary packages to have most functionality enabled by default, unless that pulls too heavy dependencies. I've always though this was the rule of thumb for ports. So my opinion is that the options mentioned in the first message should be enabled instead (except for ECW of course, for it's restricted).
Comment 15 Loïc Bartoletti 2020-02-18 13:16:45 UTC
(In reply to Dmitry Marakasov from comment #14) I'm agree with you. sunpoet@ after years what's your opinion? OPTIONS_DEFAULT= CURL EXPAT FREEXL GEOS HDF5 JASPER \ KML LIBXML2 NETCDF OPENJPEG PCRE PGSQL SFCGAL \ SPATIALITE SQLITE WEBP XERCES ZSTD POPPLER is acceptable?
Comment 16 Rainer Hurling 2020-02-18 15:40:20 UTC
(In reply to Loïc Bartoletti from comment #15) Hope it is ok that I also answer here ;) These options would be fine for our use cases with postgis, openjump, saga and qgis. Thanks Loïc, for your ongoing work.
Comment 17 Loïc Bartoletti 2020-02-19 20:27:16 UTC
Created attachment 211768 [details] GDAL 3 with options_default
Comment 18 Sunpoet Po-Chuan Hsieh 2020-03-01 11:12:35 UTC
(In reply to Dmitry Marakasov from comment #14) (In reply to Loïc Bartoletti from comment #15) (In reply to Rainer Hurling from comment #16) I plan to enable these options by default: CURL EXPAT FREEXL GEOS JASPER LIBXML2 OPENJPEG PCRE PGSQL POPPLER WEBP ZSTD Except ECW, it covers all options in comment #0.
Comment 19 Rainer Hurling 2020-03-01 16:55:41 UTC
(In reply to Sunpoet Po-Chuan Hsieh from comment #18) Hi Sunpoet, Is there any reason not to enable by default SPATIALITE, SQLITE, and SFCGAL? GDAL functionality enabled by these options is really needed by PostGIS and QGIS. Many thanks for clarification in advance, Rainer
Comment 20 Loïc Bartoletti 2020-03-02 06:07:00 UTC
Thank you! I want to emphasize these options because they are really useful. XERCES : mandatory for GML export NETCDF : mandatory for Mesh in QGIS (and part of mdal library) Spatialite/Sqlite : mandatory for GeoPackage (new lingua franca formats) and spatialite KML : mandatory for libmkl (.kml and .kmz : google earth/map exchange) SFCGAL : as you want since we're having discussions about making this build optional in postgis and its status, due to problems compiling with CGAL during upgrades. It is almost only on FreeBSD that SFCGAL/CGAL does not pose a packaging problem. Thanks again
Comment 21 Loïc Bartoletti 2020-04-29 10:17:38 UTC
Hi Sunpoet, I have some free (CPU) time next days. I can give it a try. Loïc
Comment 22 Loïc Bartoletti 2020-05-23 12:42:59 UTC
Created attachment 214779 [details] gdal 3.1.0_2 Newest version.
Comment 23 Loïc Bartoletti 2020-07-17 14:46:08 UTC
Created attachment 216516 [details] gdal 3.1.0_4 with options default
Comment 24 Nezos 2020-08-18 12:14:14 UTC
Would it be possible to include these additional options for the quarterly branch as QGIS is unusable without rebuilding gdal? This affects everyone using it as the geopackage is the standard database used by QGIS and without gdal support it does not work. I also agree with all others that gdal requires a broader scope as anyone dealing with geodata requires very often to use different kinds of files. Thank you.
Comment 25 Loïc Bartoletti 2020-10-16 04:17:13 UTC
Created attachment 218784 [details] gdal 3.1." with options default and mark postgis23 as broken Another new version updated in the ports without adding these options. After so many years and so many tickets opened, here is the last updated version. I mark postgis23 broken since gdal will use by default postgresql12, and postgis23 doesn't support a version newer than 10. I've made an exp-run on all ports that depend on gdal, py-gdal, p5-Geo-gdal, etc. Without a feedback before next week, I will commit this (before the next QGIS release). Here the list: astro/libosmium astro/merkaartor astro/qmapshack cad/sumo databases/grass7 databases/postgis23 databases/postgis24 databases/postgis25 databases/postgis30 databases/postgresql-ogr_fdw games/flightgear-terragear graphics/R-cran-rgdal graphics/gdal graphics/gmt graphics/mapcache graphics/mapcache graphics/mapserver graphics/opencv graphics/openorienteering-mapper graphics/osg graphics/osg34 graphics/osgearth graphics/p5-Geo-GDAL graphics/py-fiona graphics/py-gdal graphics/py-pyrsgis graphics/py-rasterio graphics/qgis graphics/qgis-ltr math/R-cran-sf math/pdal math/saga misc/py-kartograph
Comment 26 commit-hook 2020-10-22 12:20:57 UTC
A commit references this bug: Author: lbartoletti Date: Thu Oct 22 12:20:06 UTC 2020 New revision: 552962 URL: https://svnweb.freebsd.org/changeset/ports/552962 Log: graphics/gdal: Add more default options GDAL is an essential component for GIS software as graphics/qgis*, math/saga, databases/postgis*, databases/grass*. Unfortunately too many options are not enabled by default, causing the user of theses softwares to rebuild graphics/gdal and its bindings (especially graphics/py-gdal) for basic use. With this commit, the vast majority of user needs will be covered, leaving the build of graphics/gdal with the less relevant and less used options to the users who really need it. As gdal now requires postgresql, the package will install the default version (postgresql12), however databases/postgis23 cannot be built with postgresql>10 so it is marked as BROKEN -- however this is not a problem since it is largely deprecated and will be removed soon. PR: 224004 Approved by: Maintainer timeout Changes: head/databases/postgis23/Makefile head/graphics/gdal/Makefile
Comment 27 Loïc Bartoletti 2020-10-22 12:31:53 UTC
Comment 28 Loïc Bartoletti 2020-10-23 12:57:39 UTC
Open again. Since the commit was reverted https://svnweb.freebsd.org/ports?view=revision&revision=553000
Comment 29 Loïc Bartoletti 2020-11-18 15:59:24 UTC
Comment 30 Loïc Bartoletti 2020-11-18 16:38:27 UTC
(In reply to Loïc Bartoletti from comment #29) WTF... wrong copy/paste in bugzilla.. sorry
Comment 31 Nezos 2020-11-24 07:59:33 UTC
I don't know why Bartoletti commit was reverted but GDAL is part of OSGeo project. Therefore, options for supporting geospatial data are a must when using this library. Geospatial data without KML, SQLITE, NETCDF, PGSQL? Probably 10 years ago would be an option. If you still consider that these are not necessary then mark QGIS port as broken. It is unusable.
Comment 32 Kurt Jaeger 2020-11-24 09:13:09 UTC
(In reply to Nezos from comment #31) They were reverted, because the maintainer (sunpoet) suggests to solve the problem using subpackages. There's a review open to support subpackages: https://reviews.freebsd.org/D16457 That review is waiting for support for subpackages in poudriere, see https://github.com/freebsd/poudriere/pull/741 It needs a capable volunteer to implement it -- and as far as I understand, the FreeBSD oundation would be willing to support that. Now we try to find that capable volunteer.
Comment 33 Loïc Bartoletti 2020-11-25 05:26:20 UTC
(In reply to Kurt Jaeger from comment #32) Sorry Kurt, I respectfully disagree. Yes, using subpackages would be a solution. But, for now, we don't have subpackages in FreeBSD. And, using GIS softwares with GDAL built without this sane options, are not usable in production. One can discuss about options to be enabled by default (cf messages above) but take a look how it's packaged on other system: - https://github.com/jrmarino/ravensource/blob/master/bucket_9F/gdal/specification#L43 (ravenports) - https://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/geo/gdal/Makefile?rev=1.92&content-type=text/x-cvsweb-markup (OpenBSD have subpackages) - https://github.com/Homebrew/homebrew-core/blob/master/Formula/gdal.rb#L64 (macOS brew) - https://src.fedoraproject.org/rpms/gdal/blob/master/f/gdal.spec#_326 (Fedora) - https://github.com/archlinux/svntogit-community/blob/packages/gdal/trunk/PKGBUILD#L44 (Arch Linux) - https://salsa.debian.org/debian-gis-team/gdal/-/blob/master/debian/rules#L117 (Debian have subpackages) etc... All have more default options. There must be a reason? Yes, one can build gdal with our options via ports, but who wants to rebuild gdal, py-gdal, etc. instead of using our good pkg system? We are a small gis community in FreeBSD and how many complain about it? Quick answer: almost all... I hope this issue will be solved soon. Loïc
Comment 34 Kurt Jaeger 2020-11-25 06:31:27 UTC
(In reply to Loïc Bartoletti from comment #33) I agree with you, I would change the default as well. I just wanted to document the state of the debate in the PR. On the other hand, subpackages would be useful. But we need someone to implement it.
Comment 35 Mathieu Arnold 2020-11-25 09:48:52 UTC
Subpackages are implemented, we just need a few bits in poudriere to make it usable.
Comment 36 Kurt Jaeger 2020-11-25 10:20:29 UTC
(In reply to Mathieu Arnold from comment #35) I asked around and found no-one who would be able and willing to contribute those bits. Do you have a suggestion on who to ask ?
Comment 37 Loïc Bartoletti 2020-11-30 08:00:50 UTC
(In reply to Mathieu Arnold from comment #35) We agree, as it's not finished in poudriere yet, it's not usable and by extension for me that means it wasn't fully implemented. However, as long as this does not exist, I (we) think that the default options need to be fixed. cf https://lists.freebsd.org/pipermail/svn-ports-all/2020-October/271006.html and next messages.
Comment 38 Mathieu Arnold 2020-11-30 10:01:36 UTC
Well, yes, please make the default options set a workable set.
Comment 39 Loïc Bartoletti 2020-12-19 18:34:26 UTC
(In reply to Mathieu Arnold from comment #38) Thanks mat! I have just submitted the update of gdal 3.2.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251965. I'm waiting for the response of sunpoet@ Without a response from him, I will push this change.