Bug 244750

Summary: sysutils/py-diffoscope: ZIP comparator fails due to non-portable code
Product: Ports & Packages Reporter: Michael Osipov <michael.osipov>
Component: Individual Port(s)Assignee: Kubilay Kocak <koobs>
Status: Closed FIXED    
Severity: Affects Some People CC: michael.osipov
Priority: --- Flags: koobs: maintainer-feedback+
koobs: merge-quarterly+
Version: Latest   
Hardware: Any   
OS: Any   
URL: https://lists.reproducible-builds.org/pipermail/diffoscope/2020-March/002632.html
Attachments:
Description Flags
Patch against /usr/ports none

Description Michael Osipov 2020-03-12 10:01:32 UTC
Diffing ZIP files with zipinfo(1) exiting with 9.

The bug is fully described on the diffoscope mailing list by https://lists.reproducible-builds.org/pipermail/diffoscope/2020-March/002632.html.
It also contains a decent patch. Consider applying it until it has been upstreamed.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2020-03-12 12:00:49 UTC
Thank you for the report and upstream references Michael

Are you able to produce a patch against the port?
Comment 2 Michael Osipov 2020-03-12 12:02:56 UTC
(In reply to Kubilay Kocak from comment #1)

Yes, sure. That is straight forward. Let me work on that after lunch.
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2020-03-12 12:06:01 UTC
(In reply to Michael Osipov from comment #2)

Much appreciated :)
Comment 4 Michael Osipov 2020-03-12 13:43:11 UTC
Created attachment 212350 [details]
Patch against /usr/ports

Works for me on
> poudriere testport -j 113-release-amd64 -p dev -o sysutils/py-diffoscope -i

See
> root@113-release-amd64-dev:/usr/local # diffoscope ./lib/python3.7/test/test_importlib/zipdata01/ziptestdata.zip ./lib/python3.7/test/test_importlib/zipdata02/ziptestdata.zip
> 2020-03-12 13:36:31 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
> --- ./lib/python3.7/test/test_importlib/zipdata01/ziptestdata.zip
> +++ ./lib/python3.7/test/test_importlib/zipdata02/ziptestdata.zip
> ├── zipinfo {}
> │ @@ -1,8 +1,7 @@
> │ -Zip file size: 876 bytes, number of entries: 6
> │ +Zip file size: 698 bytes, number of entries: 5
> │  -rw-rw-r--  2.0 unx        0 b- stor 17-Dec-12 14:52 ziptestdata/__init__.py
> │ --rw-rw-r--  2.0 unx        4 b- stor 17-Dec-12 14:52 ziptestdata/binary.file
> │ --rw-rw-r--  2.0 unx       44 b- stor 17-Dec-12 14:52 ziptestdata/utf-16.file
> │ --rw-rw-r--  2.0 unx       20 b- stor 17-Dec-12 14:52 ziptestdata/utf-8.file
> │ --rw-rw-r--  2.0 unx        0 b- stor 17-Dec-12 14:52 ziptestdata/subdirectory/__init__.py
> │ --rw-rw-r--  2.0 unx        4 b- stor 17-Dec-12 14:52 ziptestdata/subdirectory/binary.file
> │ -6 files, 72 bytes uncompressed, 72 bytes compressed:  0.0%
> │ +-rw-rw-r--  2.0 unx        0 b- stor 17-Dec-12 14:52 ziptestdata/one/__init__.py
> │ +-rw-rw-r--  2.0 unx       13 b- stor 17-Dec-12 14:52 ziptestdata/one/resource1.txt
> │ +-rw-rw-r--  2.0 unx        0 b- stor 17-Dec-12 14:52 ziptestdata/two/__init__.py
> │ +-rw-rw-r--  2.0 unx       13 b- stor 17-Dec-12 14:52 ziptestdata/two/resource2.txt
> │ +5 files, 26 bytes uncompressed, 26 bytes compressed:  0.0%
> ├── zipnote {}
> │ @@ -1,19 +1,16 @@
> │  Filename: ziptestdata/__init__.py
> │  Comment:
> │
> │ -Filename: ziptestdata/binary.file
> │ +Filename: ziptestdata/one/__init__.py
> │  Comment:
> │
> │ -Filename: ziptestdata/utf-16.file
> │ +Filename: ziptestdata/one/resource1.txt
> │  Comment:
> │
> │ -Filename: ziptestdata/utf-8.file
> │ +Filename: ziptestdata/two/__init__.py
> │  Comment:
> │
> │ -Filename: ziptestdata/subdirectory/__init__.py
> │ -Comment:
> │ -
> │ -Filename: ziptestdata/subdirectory/binary.file
> │ +Filename: ziptestdata/two/resource2.txt
> │  Comment:
> │
> │  Zip file comment:
Comment 5 commit-hook freebsd_committer freebsd_triage 2020-03-13 04:46:20 UTC
A commit references this bug:

Author: koobs
Date: Fri Mar 13 04:46:14 UTC 2020
New revision: 528327
URL: https://svnweb.freebsd.org/changeset/ports/528327

Log:
  sysutils/py-diffoscope: Restore portability of zipinfo call

  /dev/stdin is a non-portable non-POSIX extension having different
  semantics on different operating systems. zininfo(1) exits with 9 when
  /dev/stdin is supplied on FreeBSD. In fact, unzip(1) explicitly documents
  that it does not support reading from stdin.

  [1] https://lists.reproducible-builds.org/pipermail/diffoscope/2020-March/002632.html

  PR:		244750
  Submitted by:	Michael Osipov <michael.osipov siemens com>
  MFH:		2020Q1

Changes:
  head/sysutils/py-diffoscope/Makefile
  head/sysutils/py-diffoscope/files/
  head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_zip.py
Comment 6 commit-hook freebsd_committer freebsd_triage 2020-03-13 04:49:21 UTC
A commit references this bug:

Author: koobs
Date: Fri Mar 13 04:48:57 UTC 2020
New revision: 528329
URL: https://svnweb.freebsd.org/changeset/ports/528329

Log:
  MFH: r528327 sysutils/py-diffoscope: Restore portability of zipinfo call

  /dev/stdin is a non-portable non-POSIX extension having different
  semantics on different operating systems. zininfo(1) exits with 9 when
  /dev/stdin is supplied on FreeBSD. In fact, unzip(1) explicitly documents
  that it does not support reading from stdin.

  [1] https://lists.reproducible-builds.org/pipermail/diffoscope/2020-March/002632.html

  PR:		244750
  Submitted by:	Michael Osipov <michael.osipov siemens com>

  Approved by:	ports-secteam (blanket: runtime bugfix)

Changes:
_U  branches/2020Q1/
  branches/2020Q1/sysutils/py-diffoscope/Makefile
  branches/2020Q1/sysutils/py-diffoscope/files/
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2020-03-13 04:49:55 UTC
Committed and merged, thank you for the report and patch Michael!
Comment 8 Michael Osipov 2020-03-13 09:10:28 UTC
(In reply to Kubilay Kocak from comment #7)

Thank you very much for the quick turnaround. I am working also other ports to make local patches obsolete and bring them upstream which I have commit permission.
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2020-03-13 09:12:44 UTC
(In reply to Michael Osipov from comment #8)

My pleasure Michael

The other thing I've had planned but not gotten around to doing is adding the various 'utilities' that the tests use (but SKIP if not available) to the ports TEST_DEPENDS.

This will likely pickup many other additional issues that will help the 'porting' process long-term

Don't hesitate to reach out if there's anything I can do to make your life easier
Comment 10 Michael Osipov 2020-03-13 09:14:41 UTC
(In reply to Kubilay Kocak from comment #9)

I have PR 244695 open which a subtile one with analysis and a patch proposal. Maybe you can have a look at it.
Comment 11 commit-hook freebsd_committer freebsd_triage 2020-03-14 02:20:45 UTC
A commit references this bug:

Author: koobs
Date: Sat Mar 14 02:20:08 UTC 2020
New revision: 528398
URL: https://svnweb.freebsd.org/changeset/ports/528398

Log:
  MFH: r526348 sysutils/py-diffoscope: Update to 136

  Merge ports r526348 (update to 136, bugfix release) which should have
  been merged with ports r528329

  PR:		244750

  Approved by:	ports-secteam (blanket: bugfix release, fix quarterly regression)

Changes:
_U  branches/2020Q1/
  branches/2020Q1/sysutils/py-diffoscope/Makefile
  branches/2020Q1/sysutils/py-diffoscope/distinfo