Bug 205852

Summary: Be nicer about multiple sqlalchemy ports
Product: Ports & Packages Reporter: Mikhail Teterin <mi>
Component: Ports FrameworkAssignee: Port Management Team <portmgr>
Status: Closed Overcome By Events    
Severity: Affects Some People CC: edacval, girgen, koobs, lwhsu, matthew, pi, ports-bugs, python, robak, tcberner, vishwin
Priority: --- Keywords: needs-qa, patch
Version: Latest   
Hardware: Any   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191442
Attachments:
Description Flags
Allow alembic and sqlalchemy-migrate to use any of the sqlalchemy ports mi: maintainer-approval? (lwhsu)

Description Mikhail Teterin freebsd_committer freebsd_triage 2016-01-03 20:59:23 UTC
I'm porting something called MediaGoblin, which requires sqlalchemy above 0.8 (it uses the new with_polimorph feature), but also sqlalchemy-migrate and alembic. Currently, sqlalchemy-0.8 is installed by databases/py-sqlalchemy-devel and creates package named sqlachlemy08

Those two list py-sqlalchemy>0.7 among dependencies, which translates into databases/py-sqlalchemy.

I'm proposing a patch, which relaxes the DEPENDS-line to accept whatever sqlalchemy is already installed on the system:

-RUN_DEPENDS=   ${PYTHON_PKGNAMEPREFIX}sqlalchemy>=0.7:${PORTSDIR}/databases/py-sqlalchemy \
+RUN_DEPENDS=   ${PYTHON_PKGNAMEPREFIX}sqlalchemy*>=0.7:${PORTSDIR}/databases/py-sqlalchemy \

Although my patch addresses only the two ports mentioned (maintainer CC-ed), it is very likely, that most (if not all) of the other ports needing sqlalchemy could do the same thing -- which is why I'm calling portmgr's attention to this ticket.
Comment 1 Mikhail Teterin freebsd_committer freebsd_triage 2016-01-03 21:02:31 UTC
Created attachment 165018 [details]
Allow alembic and sqlalchemy-migrate to use any of the sqlalchemy ports
Comment 2 Antoine Brodin freebsd_committer freebsd_triage 2016-01-03 21:31:31 UTC
This patch is wrong,  ${PYTHON_PKGNAMEPREFIX}sqlalchemy* could match ${PYTHON_PKGNAMEPREFIX}sqlalchemy-migrate for instance.
Comment 3 Mikhail Teterin freebsd_committer freebsd_triage 2016-01-03 23:35:50 UTC
(In reply to Antoine Brodin from comment #2)
> could match ${PYTHON_PKGNAMEPREFIX}sqlalchemy-migrate for instance.

Yes, indeed... The desired result can not be expressed with a glob-pattern -- only with a regex. Fortunately, a port can overwrite PKG_INFO -- even if this makes things messier, than I was hoping for.

If PKG_INFO can not be made to use "-x" instead of "-g" globally, how about something like this for the ports in question:

--- Makefile    (revision 405181)
+++ Makefile    (working copy)
@@ -11,9 +11,10 @@
 
 LICENSE=       MIT
 
-RUN_DEPENDS=   ${PYTHON_PKGNAMEPREFIX}sqlalchemy>=0.7.3:${PORTSDIR}/databases/py-sqlalchemy \
+RUN_DEPENDS=   ${PYTHON_PKGNAMEPREFIX}sqlalchemy[0-9]*$$>=0.7.3:${PORTSDIR}/databases/py-sqlalchemy \
                ${PYTHON_PKGNAMEPREFIX}mako>=0:${PORTSDIR}/textproc/py-mako
 
+PKG_INFO=      ${PKG_BIN} info -x
 USES=          python
 USE_PYTHON=    distutils autoplist
Comment 4 Antoine Brodin freebsd_committer freebsd_triage 2016-01-04 06:44:44 UTC
Something like this may work for both sqlalchemy and sqlalchemy08  (but the package builders will keep using sqlalchemy):

${PYTHON_SITELIBDIR}/sqlalchemy/__init__.py:${PORTSDIR}/databases/py-sqlalchemy

There may be side effects of overwriting PKG_INFO.
Comment 5 Mikhail Teterin freebsd_committer freebsd_triage 2016-01-04 13:02:03 UTC
(In reply to Antoine Brodin from comment #4)
> Something like this may work for both sqlalchemy and sqlalchemy08
> (but the package builders will keep using sqlalchemy):

Yes, that's an alternative -- for the particular case of sqlalchemy. But something more comprehensive may be in order. So far we've been resolving such cases with USES-foo, but it may not be warranted in all cases.

> There may be side effects of overwriting PKG_INFO

Globally? Yes. Certainly a full test-run would be needed...
Comment 6 Mikhail Teterin freebsd_committer freebsd_triage 2016-01-11 06:05:28 UTC
BTW, why can't we have multiple ports install the different versions of the same package? In this case, both py-sqlalchemy-devel and py-sqlalchemy would install py27-sqlalchemy-VERSION ... What's wrong with that?
Comment 7 Palle Girgensohn freebsd_committer freebsd_triage 2016-02-02 16:27:59 UTC
(In reply to Mikhail Teterin from comment #6)

You would have to patch every python script to import sqlalchemyNN?
Comment 8 Mikhail Teterin freebsd_committer freebsd_triage 2016-02-02 16:29:52 UTC
(In reply to Palle Girgensohn from comment #7)
The name of the Python module is still sqlachemy for all, is not it? The version may be different, but most users of the module do not care...
Comment 9 Palle Girgensohn freebsd_committer freebsd_triage 2016-02-02 16:37:27 UTC
(In reply to Mikhail Teterin from comment #8)
Umm, I don't really know python all that well, I just want some software installed... :-)

How on earth will it get the right module then?

Doesn't 

from NNN import YY

translate to importing something like

lib/python2.7/site-packages/NNN/YY.py

?
Comment 10 Palle Girgensohn freebsd_committer freebsd_triage 2016-02-02 16:41:47 UTC
But the root problem here is that we believe all the old ports will fail if we update sqlalchemy to a current version?

I also need a fresh version, in my case sqlalchemy>=1.0.

I would prefer we just update the py-sqlalchemy port. It is very old?

How much will break? Anyone knows?
Comment 11 Mikhail Teterin freebsd_committer freebsd_triage 2016-02-02 16:43:03 UTC
(In reply to Palle Girgensohn from comment #9)
This PR is not about multiple versions of SQLAlchemy coexisting on the same system (though that's possible in Python world too).

This PR is about ports, that need SOME version of SQLAlchemy, being satisfied with ANY version of SQLAlchemy, that happens to already be installed on the machine.
Comment 12 Palle Girgensohn freebsd_committer freebsd_triage 2016-02-02 16:44:02 UTC
(In reply to Mikhail Teterin from comment #11)
Why would you want that?
Comment 13 Palle Girgensohn freebsd_committer freebsd_triage 2016-02-02 16:47:22 UTC
(In reply to Palle Girgensohn from comment #12)
Ah, because there is a -devel and a standard port?

And your port will work fine with either?
Comment 14 Mikhail Teterin freebsd_committer freebsd_triage 2016-02-02 17:12:41 UTC
(In reply to Palle Girgensohn from comment #13)
> And your port will work fine with either?

No, my port wants a feature only offered by the newer version. But other ports would work fine with the newer version too -- yet, they insist on the older version.

It is this insistence, that I wish to see corrected. The insistence is recorded in the Makefiles of the ports, not in the vendors' Python-scripts.
Comment 15 Palle Girgensohn freebsd_committer freebsd_triage 2016-02-02 17:31:18 UTC
(In reply to Mikhail Teterin from comment #14)

OK. I would prefer we just had one sqlalchemy port, and bend the other ports to work with the new fresh version only.

Just for sport, I got a list of all the ports requiring sqlalchemy and I'm building them all in poudriere: 

make quicksearch key=sqlalchemy |grep ^Path | cut -f7- -d/                
biology/pycogent
databases/py-Elixir
databases/py-alembic
databases/py-flask-sqlalchemy
databases/py-geoalchemy
databases/py-geoalchemy2
databases/py-sqlalchemy
databases/py-sqlalchemy-devel
databases/py-sqlalchemy-migrate
databases/py-sqlalchemy06
deskutils/gourmet
deskutils/griffith
devel/buildbot
games/anki
games/childsplay
games/pyfa
irc/py-limnoria
multimedia/py-openlp
science/py-obspy
www/geeknote
www/py-flexget
www/py-formalchemy
www/py-imdbpy
www/py-rhodecode
www/py-turbogears2


Then I will try updating the sqlalcemy port and see who fails.

As you say, if it the Makefiles that require old versions, perhaps just kick the Makefiles in the right direction is the best option? But I expect to be wrong here, some ports will definitely fail...
Comment 16 Mikhail Teterin freebsd_committer freebsd_triage 2016-02-02 17:37:56 UTC
(In reply to Palle Girgensohn from comment #15)
> OK. I would prefer we just had one sqlalchemy port, and bend the other
> ports to work with the new fresh version only.

I very strongly prefer that too, but it just is not always achievable -- today or in the future there may be a need to retain multiple versions of a port in the tree because the latest version breaks compatibility for some of the consumers.

Such a situation should not cause more damage than necessary, however, and a port that could use multiple versions of a dependency needs to be able to say so -- instead of insisting on only one of the alternatives.
Comment 17 Kubilay Kocak freebsd_committer freebsd_triage 2016-02-02 17:40:49 UTC
(In reply to Palle Girgensohn from comment #15)

I don't believe that's feasible.

The reason why multiple versions exist because API compatibility changes between minor versions (perfectly ok), and consuming packages lag behind the latest major.minor until they have a chance to update.

The pattern of creating a new fooXY and leaving dependent ports on their older versions is because correctly, accurately and completely testing for not just build time fixes, but correct runtime function (and/or regressions) is not something a sweeping change can achieve, and is best left to individual maintainers.

Yes, maintainers should be encouraged to update the dependencies.
Yes, if older versions are no longer supported, they should be DEPRECATED. 
Yes, if port upstreams have 'bumped' their sqlalchemy dependencies, the ports should follow suite.

See Also:

Upgrade databases/py-sqlalchemy
https://reviews.freebsd.org/D908

Which is still open (CC'ing bartek@)

This is a very similar problem space to our djangoXY ports, and deserves careful consideration.

Also why is this a Ports Framework issue, when the changes are for two individual ports? 

I believe this needs python@ too, and it would be beneficial if any solution applied to other instances where multiple fooXY and multiple consumers for said ports exist, with ranged or limited dependency specifications.
Comment 18 Palle Girgensohn freebsd_committer freebsd_triage 2016-02-03 09:18:18 UTC
(In reply to Kubilay Kocak from comment #17)
Of course, you're right. The API changes need attention from each and every maintainer. But 0.7 is really their oldest release still supported. For me, I need 1.0 for my port... 

So the preferred way would be, as already stated before: 

sqlalchemy07
sqlalchemy08
sqlalchemy09
sqlalchemy   (1.0)

right?


This PR original question is about allowing a port, that is agnostic to these API changes, to use any of the above sqlalchemy ports if it is already installed (as a dependecny of a more picky port). Suggestions using globs or regex, or simply depend on a file, will all break package dependencies. I don't think that is a very good idea, but maybe I'm being overly conservative here?

PR#191442 discusses actually updating sqlalchemy. Maybe that PR is more to the point from my perspective.


Anyway, all I wany is a sqlalchemy-1.0 port that I can depend upon. My python knowledge is very shallow, and hence my interest here lies not with python packages as such. But I'm glad to help if we can get this sqlalchemy bit cleaned up a bit. :) Just tell me what I can do. I have a poudriere setup for example, but I guess most of you do. :-)
Comment 19 Matthew Seaman freebsd_committer freebsd_triage 2016-05-22 11:13:18 UTC
After r415627 (PR 191442, D908) there are now 5 sqlalchemy ports:

     Port                 Version
     ----------------------------
     py-sqlalchemy06      0.6.9
     py-sqlalchemy07      0.7.10
     py-sqlalchemy08      0.8.7
     py-sqlalchemy09      0.9.10
     py-sqlalchemy10      1.0.13

This should allow girgen to proceed with his work, and in principle it permits writing a glob expression '${PYTHON_PKGNAMEPREFIX}sqlalchemy[01][67890]' to match only the SQLAlchemy ports as suggest by Mikhail, but on the whole I think simply reviewing each of the ports that depends on sqlalchemy and having them depend on the latest possible version would be the way to go.
Comment 20 Charlie Li freebsd_committer freebsd_triage 2018-01-11 13:34:21 UTC
Has there been any further progress on this?

SQLAlchemy 1.2 has been released for two weeks now, and the 1.0 series has dropped off the cheeseshop (only the last three 1.1 series releases and 1.2.0 release and beta remain).
Comment 21 Tobias C. Berner freebsd_committer freebsd_triage 2022-09-07 15:22:22 UTC
Moin moin 

Closing this, due to old age. Please re-open the issue, if it is still relevant.


mfg Tobias