Bug 233726 - graphics/qgis: Update to 3.4.3
Summary: graphics/qgis: Update to 3.4.3
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Tobias C. Berner
URL:
Keywords:
Depends on: 233715 233716 233717
Blocks:
  Show dependency treegraph
 
Reported: 2018-12-02 17:39 UTC by Rainer Hurling
Modified: 2019-02-10 10:13 UTC (History)
5 users (show)

See Also:


Attachments
patch to update QGIS from 2.18.26 to 3.4.2 (325.91 KB, patch)
2018-12-02 17:39 UTC, Rainer Hurling
rhurlin: maintainer-approval+
Details | Diff
patch to update QGIS from 2.18.26 to 3.4.2, with more functionality (465.43 KB, patch)
2018-12-12 20:54 UTC, Rainer Hurling
rhurlin: maintainer-approval+
Details | Diff
patch to update QGIS from 2.18.26 to 3.4.3, previous suggestions included (464.82 KB, patch)
2018-12-28 15:43 UTC, Rainer Hurling
rhurlin: maintainer-approval+
Details | Diff
patch to update QGIS from 2.18.26 to 3.4.3, CMAKE_ARGS simplified (464.75 KB, patch)
2018-12-29 13:20 UTC, Rainer Hurling
rhurlin: maintainer-approval+
Details | Diff
patch to update QGIS from 2.18.26 to 3.4.3, better workaround for PyQt sip (464.55 KB, patch)
2019-01-03 10:51 UTC, Rainer Hurling
rhurlin: maintainer-approval+
Details | Diff
patch to update QGIS from 2.18.26 to 3.4.3, renewed after QGIS 2 patch (464.77 KB, patch)
2019-01-07 21:03 UTC, Rainer Hurling
rhurlin: maintainer-approval+
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Rainer Hurling 2018-12-02 17:39:57 UTC
Created attachment 199758 [details]
patch to update QGIS from 2.18.26 to 3.4.2

This is a major update from QGIS version 2 to 3 (2.18.26 to 3.4.2).

After QGIS 2 is no longer the LTR version[1] and QT4 in the ports tree is marked for removal on 2019-03-15 (r486467), it is time to introduce the long awaited QGIS 3 port on FreeBSD ;)


Changes of the port:
- Update to 3.4.2
- Change from QT4 to QT5
- Change from Python 2.7 to Python 3.6
- Change and/or update several dependencies
- Add and delete several Python modules
- Reorder CMAKE_ARGS
- Update patches under files
- Update pkg-plist


======================================================
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Before commit this update, please commit the following
PRs, they are showstoppers for QGIS3:

bug #233715, bug #233716, bug #233717, and bug #233718

In addition, the QGIS 3 patch already has the upcoming
GRASS 7.4.3 (instead of 7.4.1) as dependency.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================


The patch is tested on Poudriere (11.2i/a, 12.0RC3i/a, and HEADi/a),
'portlint -AC' seems happy.


[1] https://qgis.org/en/site/
Comment 1 Rainer Hurling 2018-12-12 20:54:37 UTC
Created attachment 200068 [details]
patch to update QGIS from 2.18.26 to 3.4.2, with more functionality

This second, reworked update of QGIS from 2.18.26 to 3.4.2 brings, additionally to the initial patch, several more improvements.

Changes of the port are now:
- Update to 3.4.2
- Change from QT4 to QT5
- Change from Python 2.7 to Python 3.6
- Always build PYTHON and POSTGIS
- Remove DEVELOPER group
- Always build CUSTOMWIDGETS, ASTYLE, and TESTS
- Add 3D functionality
- correct time_t for i386
- Change and/or update several dependencies
- Add and delete several Python modules
- Reorder and simplify CMAKE_ARGS
- Add some FreeBSD upstream code
- Update patches under files
- Update pkg-plist


Other than stated in the first patch, it is _not_ necessary to commit bug #233715, bug #233716, bug #233717, and bug #233718 before the QGIS3 update. It turned out, that those PR's likely have problems with a first install via portmaster.


The patch is tested on Poudriere (11.2i/a, 12.0i/a, and HEADi/a),
'portlint -AC' seems happy.
Comment 2 Tobias C. Berner freebsd_committer 2018-12-13 05:25:41 UTC
Moin moin

Thanks for the patch :) 


I noticed some issues when looking at it:

-DQSCI_SIP_DIR:PATH=${LOCALBASE}/share/PyQt5/3.6/sip/Qsci 
this should probably be 
-DQSCI_SIP_DIR:PATH=${PYQT_APIDIR}

and this
-DSIP_DEFAULT_SIP_DIR:FILEPATH=${LOCALBASE}/share/PyQt5/3.6/sip
should be
-DQSCI_SIP_DIR:PATH=${PYQT_SIPDIR}



Instead of writing
CMAKE_ARGS= -DWITH_3D:BOOL=TRUE 
you can do
CMAKE_ON=  WITH_3D
respectively 
CMAKE_OFF=  WITH_QWTPOLAR



Instead of
GRASS_CMAKE_ON=		-DWITH_GRASS7:BOOL=TRUE
GRASS_CMAKE_OFF=	-DWITH_GRASS7:BOOL=FALSE
you could write 
GRASS_CMAKE_BOOL=   WITH_GRASS7


* Your pre-configure looks like something for BINARY_ALIAS, or shebangfix
* Your post-configure targets should again use the ${OYQT_APIDIR}, otherwise it will only work for python-3.6.


* patch-cmake_PyQtMacros.cmake is wrong :D 




mfg Tobias
Comment 3 Loïc Bartoletti 2018-12-14 11:00:09 UTC
To avoid the post-configure patch, you should edit cmake/FindPyQt5.py, with something like:
              os.path.join(sip_dir, "PyQt5", str(sys.version_info.major) + "." + str(sys.version_info.minor)),

I had already patched this upstream file to find our installation, I think this new addition is necessary since our change in pyqt?

Let me know if it works well and I will modify upstream.

Thanks
Comment 4 Tobias C. Berner freebsd_committer 2018-12-14 15:38:48 UTC
(In reply to lbartoletti from comment #3)
From PyQt5 5.11 on there is a "private" PyQt5 sip module. 
I would say, build it against pr 233954 which updates to a more recent version, which will probably make it working without any patches.


mfg Tobias
Comment 5 Loïc Bartoletti 2018-12-18 12:36:55 UTC
Some additional comments:

- ICON_SIZES is no longer needed
- patch-cmake_PyQtMacros.cmake is not needed but you have to add something like CMAKE_ARGS+= -DPYRCC_PROGRAM:STRING="${LOCALBASE}/bin/pyrcc5-${PYTHON_VER}"

Thanks Rainer
Comment 6 Loïc Bartoletti 2018-12-18 12:39:59 UTC
(In reply to Tobias C. Berner from comment #2)

> * Your pre-configure looks like something for BINARY_ALIAS, or shebangfix

I have patched all py files upstream to avoid "/usr/bin/xx  to /usr/bin/env xx"

But, on FreeBSD if you install python3 via "pkg install python3" a symlink for python3 is created. On the other hand, if you install python3.6 via another command, python3 is missing, only python3.6 is available. I think Rainer does this to get around this link problem?
Comment 7 Rainer Hurling 2018-12-18 13:04:30 UTC
(In reply to lbartoletti from comment #6)

Hi Loïc, hi Tobias,

First, I have to apologize for the long response time. In the last days, I was busy with updating my system and Poudriere with gcc8, qt5 etc. Tobias, your patches for PyQt5 are really appreciated.

Yes, Loïc is right about my decision, to patch against the minor version of Python, here 3.6. If you install for example via Portmaster, the links /usr/local/bin/python3 and/usr/local/bin/python3-config are not generated. And there are many reasons to configure and build QGIS and deps for your own (i.e. gdal is missing several formats with default settings) ;)

ATM, I pick up your both suggestions to simplify and generalize the port some more. I hope to be ready with testing in the next couple of days.
Comment 8 Loïc Bartoletti 2018-12-18 15:27:02 UTC
I'm trying to compile your port and I realize that I'm going to have to patch grass and txt2tag because QGIS wants python3 and they are in python2. On a fresh installation, I saw errors like "xxx wants Python3... and flavors is Python2"
Comment 9 Loïc Bartoletti 2018-12-20 06:06:28 UTC
(In reply to Tobias C. Berner from comment #4)

I tested with the new version of PyQt in the ports (nice work by the way;)) and it is still necessary to have the patch to find the.sip files

I think the problem comes from our sip configuration. If I do this in Python (From ${WRKSRC}/cmake/FindSIP.py):

import sipconfig
sipcfg = sipconfig.Configuration()
sipcfg.default_sip_dir

I get /usr/local/share/PyQtsip/3.6/sip that doesn't exist

It seems to me we've already had a problem like this, but I couldn't find the related ticket.
Comment 10 Rainer Hurling 2018-12-20 06:15:15 UTC
(In reply to lbartoletti from comment #9)

Hi Loïc,

That's exactly the problem, I am stumbling about the last days. Because of the wrong path, the build process does not find QtXml/QtXmlmod.sip. That was the reason, I used my "hardcoded" patches in the previous version.
Comment 11 Rainer Hurling 2018-12-28 15:43:42 UTC
Created attachment 200580 [details]
patch to update QGIS from 2.18.26 to 3.4.3, previous suggestions included

This patch includes the comments and suggestions of T.C.Berner and L.Bartoletti, many thanks for them.

After some more adaptions and testing, especially for the new cmake behaviour with outsourcing, I decided to also take the next step and update QGIS to version 3.4.3. 

Changes of the port:
- Update to 3.4.3
- Remove deprecation and expiration date
- Change from QT4 to QT5
- Change from Python 2.7 to Python 3.6
- Always build PYTHON and POSTGIS
- Remove DEVELOPER group
- Always build CUSTOMWIDGETS, ASTYLE, and TESTS
- Add 3D functionality
- correct time_t for i386
- Change and/or update several dependencies
- Add and delete several Python modules
- Change to cmake:outsource functionality
- Reorder CMAKE_ARGS
- Update patches under files
- Update pkg-plist


Bug #233715, bug #233716, bug #233717, and bug #233718 are not necessary for this update. It turned out, that those PR's more likely only have problems with a first install via portmaster. This has nothing to do with QGIS and the port.


The patch is tested on Poudriere (11.2i/a, 12.0i/a, and HEADi/a),
'portlint -AC' seems happy.
Comment 12 Tobias C. Berner freebsd_committer 2018-12-28 17:53:26 UTC
Comment on attachment 200580 [details]
patch to update QGIS from 2.18.26 to 3.4.3, previous suggestions included

I would rewrite the CMAKE_ARGS a bit. The boolean ones you can write much more readable with CMAKE_ON and CMAKE_OFF.
Comment 13 Rainer Hurling 2018-12-29 13:20:33 UTC
Created attachment 200600 [details]
patch to update QGIS from 2.18.26 to 3.4.3, CMAKE_ARGS simplified

Playing around with CMAKE_ON and CMAKE_OFF could be something like in the renewed patch. Thanks for the tip.
Comment 14 Loïc Bartoletti 2019-01-02 10:42:37 UTC
rainer, tcberner,

A suggestion for sip. Instead of having patches in post-configure. I propose to replace https://github.com/qgis/QGIS/blob/master/cmake/FindPyQt5.py#L57 in that way:

'pyqt_sip_dir': sip_dir.replace('PyQtsip', 'PyQt5'),

Also I think that
"-DSIP_DEFAULT_SIP_DIR:FILEPATH=${PYQT_SIPDIR}" has no effects

Regards
Comment 15 wen 2019-01-03 07:23:31 UTC
Hi, all:

  Since QGIS-2.18 is an old LTR version, I hope before update to 3.4 we should repocopy it to qgis2.
  How about it ?

wen
Comment 16 Tobias C. Berner freebsd_committer 2019-01-03 08:40:31 UTC
(In reply to wen from comment #15)
qt:4 and therewith that repocopy will be removed in march... Do you think thst is still worth it?
Comment 17 Rainer Hurling 2019-01-03 10:51:48 UTC
Created attachment 200737 [details]
patch to update QGIS from 2.18.26 to 3.4.3, better workaround for PyQt sip

Thanks, Loïc, for the suggestions in comment #14. This is a much more general solution for the workaround with PyQt5 sip. Really appreciated.

And yes, it seems that "-DSIP_DEFAULT_SIP_DIR:FILEPATH=${PYQT_SIPDIR}" has no effects. So I removed it.

A new patch should contain these changes.


The patch is tested on Poudriere, 'portlint -AC' seems happy.
Comment 18 Rainer Hurling 2019-01-03 11:02:15 UTC
(In reply to wen from comment #15 and comment #16)

Hi Wen, hi Tobias, I discussed this before with Loïc and we were undecided what would be the best practise. QGIS-2.18.28 LTR will be the last release of the QGIS 2.x branch. If we would repocopy QGIS2, I probably would also update the port (graphics/qgis2 ?) a last time from version 2.18.26 to 2.18.28 (upcoming release).

A repocopy could be very important for long time users, because there are so many changes in QGIS3. A change from QGIS2 to QGIS3 is often very tricky for existing projects and will need some time.

If we would repocopy, a big warning about the end of QT4 in march is also needed in the QGIS2 port ...
Comment 19 Loïc Bartoletti 2019-01-04 05:30:48 UTC
Hi,

Keeping QGIS2 in ports is a real issue.
On the one hand, QGIS2 will no longer be maintained from February 2019 and is based on a library (QT4) that has not been maintained since 2015 and will be deleted on FreeBSD in March 2019.
On the other hand, QGIS3 is now the LTR, it is usable at home (I work on it daily) and is based on the latest libraries.

So why keep QGIS 2? I often hear these arguments:
- There are still some missing plugins on QGIS 3
- There are problems or regressions on QGIS 3

I know that QGIS 2 will continue to be used for some time (2-3 years?). Which I regret.

However, I find that this is not enough to keep QGIS 2 in FreeBSD ports. Did you imagine maintaining all the dependencies? Look at the graph https://github.com/lbartoletti/lbartoletti.github.io/blob/master/qgis.svg.

As a personal matter, since I will have to continue to make some further developments on QGIS 2, I have made the choice to use a Windows VM (this is where QGIS 2 clients are located) and to keep a jail with an installation of QGIS 2 on FreeBSD 11.2. However, I can only encourage you to migrate to QGIS 3:)
Comment 20 Rainer Hurling 2019-01-06 08:22:15 UTC
After having some more time to think about it, I believe that it would be better to replace the QGIS 2 port by QGIS 3, without having a graphics/qgis2 legacy port. I personally will not have the time to also maintain a QGIS 2 port with increasing problems of its dependencies.

If we proceed in this way, it would be important to have some hints for users about the situation with dependencies being omitted. Also a big warning seems necessary, that a change to QGIS 3 will eventually break some of their existing projects and should be well prepared.

What do you think about it?
Comment 21 Tobias C. Berner freebsd_committer 2019-01-06 12:01:04 UTC
(In reply to Rainer Hurling from comment #20)

I think it is the right choice. As Qt4 will be gone from ports in 2 months.

Can you prepare an entry for the UPDATING file, with hints to the users?


mfg Tobias
Comment 22 Rainer Hurling 2019-01-06 14:40:56 UTC
Hi Loïc, hi Tobias,

----------------------------------
What do you think about the following draft of an entry for the UPDATING file:


Attention please to users of graphics/qgis (version 2.x)!

QGIS 3.4 LTR, the first long-term release (LTR) of QGIS 3, has landed in the FreeBSD ports tree. It will also replace the previous 2.18 LTR in the QGIS package repositories in February 2019 [1].

Beside, the latest QT4 version 4.8.7 has been EOL since December 2015 and its expiration date on FreeBSD is set to March, 15th 2019. Because QGIS 2 is based on QT4 and Python2, it was decided to not create a legacy port for QGIS 2, after graphics/qgis changed to QGIS 3.

For users moving over from the 2.18 LTR there is a huge list of new features and impactful changes in this new 3.4 LTR version. Most notably 3.x plugins are incompatible with 2.x plugins, so please review your plugin usage carefully. Before moving over, if you have not already done so, take a look at the changelogs from versions 3.0, 3.2, and 3.4 [2][3][4] to understand the full scope of changes in the 3.x release.


[1] https://qgis.org/en/site/getinvolved/development/roadmap.html#release-schedule
[2] http://changelog.qgis.org/en/qgis/version/3.0.0/
[3] http://changelog.qgis.org/en/qgis/version/3.2.0/
[4] http://changelog.qgis.org/en/qgis/version/3.4-LTR/
----------------------------------


Please feel free, to correct this draft, thanks in advance.
Comment 23 wen 2019-01-07 07:54:15 UTC
Hi,all:

   I did not notice that qt4 should be removed from portstree in 2 months.
   So I agree there is no need to keep qgis2 now.

   Hope this PR committed soon, I am waiting for it.

wen
Comment 24 Loïc Bartoletti 2019-01-07 08:39:51 UTC
For all those who will be interested, I have created an image of a jail with QGIS 2.

Rainer, it's OK for your message.

In addition, I noticed that processing uses numpy and as usual it is a problem with gcc. Either do the mapping as indicated in the GRASS pkg-message, or "setenv LD_LIBRARY_PATH /usr/local/lib/gcc8".

Thanks.
Comment 25 Rainer Hurling 2019-01-07 12:12:22 UTC
(In reply to lbartoletti from comment #24)

Sorry Loïc, but I do not fully understand. Yes, py-numpy is needed for processing tools and plugins. On my boxes with HEAD amd64 it seems to me, that GRASS works fine from within QGIS 3.

What exactly do you mean by either (1) do the mapping as indicated in the GRASS pkg-message, or (2) "setenv LD_LIBRARY_PATH /usr/local/lib/gcc8"?  Unfortunately, I have no experience with this.
Comment 26 Loïc Bartoletti 2019-01-07 12:31:45 UTC
Rainer,

This is my error message on my dev machine (not from your port):
Couldn't load plugin 'processing' 

ImportError: Importing the multiarray numpy extension module failed.  Most likely you are trying to import a failed build of numpy. If you're working with a numpy git repo, try `git clean -xdf` (removes all files not under version control).  Otherwise reinstall numpy. Original error was: /lib/libgcc_s.so.1: version GCC_4.8.0 required by /usr/local/lib/gcc8/libgfortran.so.5 not found 

Maybe I'm missing something.
Comment 27 Rainer Hurling 2019-01-07 14:22:00 UTC
(In reply to lbartoletti from comment #26)

Loïc,

I do not have this error with numpy for some time now. Also, if I start QGIS, all seems fine with numpy.

In my port for QGIS 3, it was important to have USES=fortran because of numpy, GRASS etc. Perhaps, this is missing in your dev port?
Comment 28 Tobias C. Berner freebsd_committer 2019-01-07 18:41:02 UTC
Moin moin

Can you regenerate the patch against the current tree?


mfg Tobias
Comment 29 Rainer Hurling 2019-01-07 21:03:18 UTC
Created attachment 200886 [details]
patch to update QGIS from 2.18.26 to 3.4.3, renewed after QGIS 2 patch

Moin moin,

Yes, of course. Here is a patch against graphics/qgis (version 2, r489603).
Comment 30 Loïc Bartoletti 2019-01-15 12:30:58 UTC
Rainer,

Normally, with https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=223409#c5 you won't have to patch sip.

Regards
Comment 31 Rainer Hurling 2019-01-15 19:08:48 UTC
(In reply to lbartoletti from comment #30)

Thanks Loïc, for the hint.

I tested graphics/qgis 3 with a patched version of devel/py-sip [1] and it builds fine without my patch files/patch-cmake_FindPyQt5.py. So, in case [1] lands before QGIS 3 will be committed, we would not need patch-cmake_FindPyQt5.py anymore.

[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=223409#c5
Comment 32 Tobias C. Berner freebsd_committer 2019-01-15 21:07:21 UTC
(In reply to Rainer Hurling from comment #31)
I think I will commit qgis in 5minutes :) -- with some some additional shebangs-fixes, so that stage-qa is happy :)

mfg Tobias
Comment 33 commit-hook freebsd_committer 2019-01-15 21:31:13 UTC
A commit references this bug:

Author: tcberner
Date: Tue Jan 15 21:30:18 UTC 2019
New revision: 490430
URL: https://svnweb.freebsd.org/changeset/ports/490430

Log:
  graphics/qgis: update to 3.4.3

  QGIS 3.4 LTR, the first long-term release (LTR) of QGIS 3.

  This is a major upgrade and users should consult the UPDATING entry
  provided for further information on how to progress.

  http://changelog.qgis.org/en/qgis/version/3.4-LTR/

  PR:		233726
  Submitted by:	Rainer Hurling <rhurlin@gwdg.de> (maintainer)

Changes:
  head/UPDATING
  head/graphics/qgis/Makefile
  head/graphics/qgis/distinfo
  head/graphics/qgis/files/patch-cmake_FindPyQt5.py
  head/graphics/qgis/files/patch-debian_qgis.desktop
  head/graphics/qgis/files/patch-python_plugins_processing_algs_saga_SagaAlgorithmProvider.py
  head/graphics/qgis/files/patch-python_plugins_processing_algs_saga_SagaUtils.py
  head/graphics/qgis/files/patch-src_app_main.cpp
  head/graphics/qgis/files/patch-src_core_qgsproject.cpp
  head/graphics/qgis/files/patch-src_plugins_grass_CMakeLists.txt
  head/graphics/qgis/files/patch-src_plugins_grass_qtermwidget_kpty.cpp
  head/graphics/qgis/pkg-descr
  head/graphics/qgis/pkg-plist
Comment 34 Tobias C. Berner freebsd_committer 2019-02-10 10:13:16 UTC
Seems there were no issues, so time to close it -- Thanks for the patches :)