Bug 243669

Summary: lang/python37 : Fails to upgrade 'python37-3.7.5_1' to 'python37-3.7.6' on FreeBSD 12.1-RELEASE-p1 powerpc 32 bit: Grammar3.7.6.final.0.pickle:No such file or directory
Product: Ports & Packages Reporter: canardo <canardo909>
Component: Individual Port(s)Assignee: freebsd-python (Nobody) <python>
Status: Open ---    
Severity: Affects Only Me CC: pkubaj, powerpc, python, wen
Priority: --- Keywords: needs-qa
Version: LatestFlags: koobs: maintainer-feedback? (python)
Hardware: powerpc   
OS: Any   
Attachments:
Description Flags
poudriere log
none
FreeBSD.bug.243669.portupgrade.python37.failed-FULL.LOG none

Description canardo 2020-01-28 11:10:58 UTC
Problem found with lang/python37 when Upgrading Upgrading 'python37-3.7.5_1' to 'python37-3.7.6' on FreeBSD 12.1-RELEASE-p1 powerpc 32 bit, running on Apple Powerbook 17".


# portupgrade -R --batch python37
[Reading data from pkg(8) ... - 1209 packages found - done]
[Gathering depends for lang/python37 .................. done]
[Exclude up-to-date packages ......... done]
--->  Upgrading 'python37-3.7.5_1' to 'python37-3.7.6' (lang/python37)
--->  Building '/usr/ports/lang/python37'
===>  Cleaning for python37-3.7.6
===>  License PSFL accepted by the user
===>   python37-3.7.6 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by python37-3.7.6 for building
===>  Extracting for python37-3.7.6
=> SHA256 Checksum OK for python/Python-3.7.6.tar.xz.
===>  Patching for python37-3.7.6
===>  Applying FreeBSD patches for python37-3.7.6
===>   python37-3.7.6 depends on file: /usr/local/lib/libncurses.so.6 - found
===>   python37-3.7.6 depends on package: pkgconf>=1.3.0_1 - found
===>   python37-3.7.6 depends on executable: msgfmt - found
===>   python37-3.7.6 depends on shared library: libffi.so - found (/usr/local/lib/libffi.so)
===>   python37-3.7.6 depends on shared library: libreadline.so.8 - found (/usr/local/lib/libreadline.so.8)
===>   python37-3.7.6 depends on shared library: libintl.so - found (/usr/local/lib/libintl.so)
===>  Configuring for python37-3.7.6
configure: loading site script /usr/ports/Templates/config.site
checking build system type... powerpc-portbld-freebsd12.1
.....
.....
.....
PYTHONPATH=/usr/ports/lang/python37/work/stage/usr/local/lib/python3.7 LD_LIBRARY_PATH=/usr/ports/lang/python37/work/Python-3.7.6  ./python -E -m lib2to3.pgen2.driver /usr/ports/lang/python37/work/stage/usr/local/lib/python3.7/lib2to3/Grammar.txt
Generating grammar tables from /usr/ports/lang/python37/work/stage/usr/local/lib/python3.7/lib2to3/Grammar.txt
Writing grammar tables to /usr/ports/lang/python37/work/stage/usr/local/lib/python3.7/lib2to3/Grammar3.7.5.final.0.pickle
PYTHONPATH=/usr/ports/lang/python37/work/stage/usr/local/lib/python3.7 LD_LIBRARY_PATH=/usr/ports/lang/python37/work/Python-3.7.6  ./python -E -m lib2to3.pgen2.driver /usr/ports/lang/python37/work/stage/usr/local/lib/python3.7/lib2to3/PatternGrammar.txt
Generating grammar tables from /usr/ports/lang/python37/work/stage/usr/local/lib/python3.7/lib2to3/PatternGrammar.txt
Writing grammar tables to /usr/ports/lang/python37/work/stage/usr/local/lib/python3.7/lib2to3/PatternGrammar3.7.5.final.0.pickle
Creating directory /usr/local/include/python3.7m
.....
.....
.....
--->  Deinstalling 'python37-3.7.5_1'
--->  Preserving /usr/local/lib/libpython3.7m.so.1.0 as /usr/local/lib/compat/pkg/libpython3.7m.so.1.0
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
	python37-3.7.5_1

Number of packages to be removed: 1

The operation will free 108 MiB.
[1/1] Deinstalling python37-3.7.5_1...
[1/1] Deleting files for python37-3.7.5_1: 100%
[Reading data from pkg(8) ... - 1208 packages found - done]
--->  Installing the new version via the port
===>  Deinstalling for python37
===>   python37 not installed, skipping
===>  Installing for python37-3.7.6
===>  Checking if python37 is already installed
===>   Registering installation for python37-3.7.6 as automatic
pkg-static: Unable to access file /usr/ports/lang/python37/work/stage/usr/local/lib/python3.7/lib2to3/Grammar3.7.6.final.0.pickle:No such file or directory
pkg-static: Unable to access file /usr/ports/lang/python37/work/stage/usr/local/lib/python3.7/lib2to3/PatternGrammar3.7.6.final.0.pickle:No such file or directory
*** Error code 74

Stop.
make[1]: stopped in /usr/ports/lang/python37
*** Error code 1

Stop.
make: stopped in /usr/ports/lang/python37
** Command failed [exit code 1]: /usr/bin/script -qa /tmp/portupgrade20200128-37053-66udaa env UPGRADE_TOOL=portupgrade UPGRADE_PORT=python37-3.7.5_1 UPGRADE_PORT_VER=3.7.5_1 make BATCH=yes -DINSTALLS_DEPENDS reinstall
--->  Restoring the old version
Installing python37-3.7.5_1...
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2020-01-28 11:19:53 UTC
Thank you for your report.

The apparent reference to 3.7.5 during the 3.7.6 build is certainly interesting:

Writing grammar tables to /usr/ports/lang/python37/work/stage/usr/local/lib/python3.7/lib2to3/Grammar3.7.5.final.0.pickle

Could you please:

 - Include the complete build log as an attachment
 - Let us know what port option are set for lang/python37
 - Confirm whether or not installation of python37 (3.7.6) works after removing (pkg delete) python37-3.7.5_1 first
Comment 2 Piotr Kubaj freebsd_committer 2020-01-28 12:27:48 UTC
Created attachment 211128 [details]
poudriere log

This seems to be because of unclean environment. I created a jail with FreeBSD 12.1 and it seems to build.
Comment 3 canardo 2020-01-28 12:37:01 UTC
(In reply to Piotr Kubaj from comment #2)

Do you have a command/script so I could clean up my environment and try again ?
Comment 4 Piotr Kubaj freebsd_committer 2020-01-28 13:02:25 UTC
(In reply to canardo from comment #3)
Well, you would need to remove all the packages to be safe, but probably removing the already installed python37 will help.
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2020-01-29 02:41:15 UTC
(In reply to Piotr Kubaj from comment #2)

Upstream expects Python to build correctly/cleanly even with existing Python installations existing in the environment. If a root cause can be identified, we should report it upstream
Comment 6 canardo 2020-01-29 16:05:49 UTC
Tests performed on my machine:

- python 3.7 was in version 3.7.5_1
- # make deinstall
- # make clean
- # make -DBATCH install clean
- python 3.7 successfully installed in version 3.7.6

Conclusions:
1) Python 3.7 in version 3.7.6 can be installed

2) From previous use few days ago, portupgrade command was successfull for some ports

1+2) It seems there's something wrong within the clean process of portupgrade, probably not specific to python3.7

If possible, next step would be to reproduce the same problem again, first on my machine.
So, I need to 
- deinstall python3.7 in version 3.7.6 => easy
- install python 3.7 in version 3.7.5_1 and try again the upgrade process, my question is: how do I do that ?
What is the command to specifically install python 3.7 version 3.7.5_1 on my machine ?


NB: please find attached the full log of the failed upgrade process.
Comment 7 canardo 2020-01-29 16:06:45 UTC
Created attachment 211170 [details]
FreeBSD.bug.243669.portupgrade.python37.failed-FULL.LOG
Comment 8 Piotr Kubaj freebsd_committer 2020-01-29 21:44:02 UTC
(In reply to canardo from comment #6)
You can use ports-mgmt/portdowngrade to downgrade a port.
Comment 9 canardo 2020-01-30 14:09:00 UTC
(In reply to Piotr Kubaj from comment #8)

Thanks for the info.
Problem is ports-mgmt/portdowngrade requires devel/subversion

And devel/subversion curently fails to build on PowerPC 32
I reported this problem last month: PR 242680

In the meanwhile, I'll keep Python 3.7.6 manually installed as it is.

If someone can systematically reproduce the upgrade problem, maybe a fix can be found. If needed, I can perform other tests on my machine.
Comment 10 Piotr Kubaj freebsd_committer 2020-01-31 12:32:35 UTC
(In reply to canardo from comment #9)
Right....

You can hack portdowngrade to use base svnlite. Or you can manually edit python37 to use version 3.7.5. Basically revert https://svnweb.freebsd.org/ports?view=revision&revision=520724.
Comment 11 Piotr Kubaj freebsd_committer 2020-04-05 18:34:54 UTC
Subversion should already work on powerpc.
Comment 12 canardo 2020-04-06 10:35:28 UTC
(In reply to Piotr Kubaj from comment #11)

Indeed subversion now build OK on powerpc (was blocked by a bug in a required dependency).