Currently there are multiple zlib variants in kernel: sys/cddl/contrib/opensolaris/uts/common/zmod/zmod.c sys/kern/subr_inflate.c sys/libkern/zlib.c There should be only one copy of zlib, ideally share the same copy of contrib/zlib.
Earlier work to consolidate zlib instances: https://svnweb.freebsd.org/base?view=revision&revision=199954
That SVN revision is too old and won't easily apply to the latest. I took a different approach and created https://reviews.freebsd.org/D19706
Assign to delphij@ for tracking, this is in progress.
A commit references this bug: Author: delphij Date: Thu May 23 05:17:19 UTC 2019 New revision: 348148 URL: https://svnweb.freebsd.org/changeset/base/348148 Log: Delete unneeded #include <sys/inflate.h> from sys/mips. PR: 229763 Submitted by: Yoshihiro Ota <ota at j.email.ne.jp> Reviewed by: imp Differential Revision: https://reviews.freebsd.org/D20190 Changes: head/sys/mips/mips/elf_trampoline.c
A commit references this bug: Author: delphij Date: Fri May 24 02:44:16 UTC 2019 New revision: 348222 URL: https://svnweb.freebsd.org/changeset/base/348222 Log: cryptodeflate: Drop z_stream zbuf.state->dummy from SDT probe. For older versions of zlib, dummy was a workaround for compilers that do not handle opaque type definition well; on FreeBSD, it's representing a value that is not really useful for monitoring purposes, and the field would be gone in newer zlib versions. PR: 229763 Submitted by: Yoshihiro Ota <ota at j.email.ne.jp> Differential Revision: https://reviews.freebsd.org/D20222 Changes: head/sys/opencrypto/cryptodeflate.c
A commit references this bug: Author: delphij Date: Fri May 24 05:34:23 UTC 2019 New revision: 348225 URL: https://svnweb.freebsd.org/changeset/base/348225 Log: Remove kgzip and kgzldr. PR: 183666, 229763 Submitted by: Yoshihiro Ota <ota at j.email.ne.jp> Differential Revision: https://reviews.freebsd.org/D20248 Changes: head/ObsoleteFiles.inc head/stand/i386/Makefile head/stand/i386/kgzldr/ head/targets/pseudo/userland/Makefile.depend head/targets/pseudo/userland/misc/Makefile.depend head/usr.sbin/Makefile.amd64 head/usr.sbin/Makefile.i386 head/usr.sbin/kgzip/
A commit references this bug: Author: delphij Date: Mon Jun 17 19:49:15 UTC 2019 New revision: 349151 URL: https://svnweb.freebsd.org/changeset/base/349151 Log: Separate kernel crc32() implementation to its own header (gsb_crc32.h) and rename the source to gsb_crc32.c. This is a prerequisite of unifying kernel zlib instances. PR: 229763 Submitted by: Yoshihiro Ota <ota at j.email.ne.jp> Differential Revision: https://reviews.freebsd.org/D20193 Changes: head/lib/libufs/Makefile head/stand/libsa/crc32_libkern.c head/sys/conf/files head/sys/dev/iscsi/icl_soft.c head/sys/dev/iscsi_initiator/isc_subr.c head/sys/dev/liquidio/lio_bsd.h head/sys/dev/usb/net/if_cdce.c head/sys/fs/ext2fs/ext2_csum.c head/sys/fs/nandfs/nandfs_segment.c head/sys/fs/nandfs/nandfs_subr.c head/sys/fs/nandfs/nandfs_vfsops.c head/sys/geom/part/g_part_bsd64.c head/sys/geom/part/g_part_gpt.c head/sys/geom/raid/md_ddf.c head/sys/kern/subr_compressor.c head/sys/libkern/crc32.c head/sys/libkern/gsb_crc32.c head/sys/libkern/x86/crc32_sse42.c head/sys/netinet/libalias/alias_sctp.c head/sys/netinet/sctp_crc32.c head/sys/netpfil/pf/pf.c head/sys/sys/gsb_crc32.h head/sys/sys/libkern.h head/sys/sys/param.h head/sys/ufs/ffs/ffs_alloc.c head/sys/ufs/ffs/ffs_snapshot.c head/sys/ufs/ffs/ffs_subr.c head/sys/ufs/ffs/ffs_vfsops.c head/tests/sys/kern/Makefile head/tests/sys/kern/libkern_crc32.c
A commit references this bug: Author: delphij Date: Thu Aug 1 06:35:36 UTC 2019 New revision: 350496 URL: https://svnweb.freebsd.org/changeset/base/350496 Log: Allow Kernel to link in both legacy libkern/zlib and new sys/contrib/zlib, with an eventual goal to convert all legacl zlib callers to the new zlib version: * Move generic zlib shims that are not specific to zlib 1.0.4 to sys/dev/zlib. * Connect new zlib (1.2.11) to the zlib kernel module, currently built with Z_SOLO. * Prefix the legacy zlib (1.0.4) with 'zlib104_' namespace. * Convert sys/opencrypto/cryptodeflate.c to use new zlib. * Remove bundled zlib 1.2.3 from ZFS and adapt it to new zlib and make it depend on the zlib module. * Fix Z_SOLO build of new zlib. PR: 229763 Submitted by: Yoshihiro Ota <ota j email ne jp> Reviewed by: markm (sys/dev/zlib/zlib_kmod.c) Relnotes: yes Differential Revision: https://reviews.freebsd.org/D19706 Changes: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c head/sys/cddl/contrib/opensolaris/uts/common/zmod/adler32.c head/sys/cddl/contrib/opensolaris/uts/common/zmod/crc32.h head/sys/cddl/contrib/opensolaris/uts/common/zmod/deflate.c head/sys/cddl/contrib/opensolaris/uts/common/zmod/deflate.h head/sys/cddl/contrib/opensolaris/uts/common/zmod/inffast.c head/sys/cddl/contrib/opensolaris/uts/common/zmod/inffast.h head/sys/cddl/contrib/opensolaris/uts/common/zmod/inffixed.h head/sys/cddl/contrib/opensolaris/uts/common/zmod/inflate.c head/sys/cddl/contrib/opensolaris/uts/common/zmod/inflate.h head/sys/cddl/contrib/opensolaris/uts/common/zmod/inftrees.c head/sys/cddl/contrib/opensolaris/uts/common/zmod/inftrees.h head/sys/cddl/contrib/opensolaris/uts/common/zmod/opensolaris_crc32.c head/sys/cddl/contrib/opensolaris/uts/common/zmod/trees.c head/sys/cddl/contrib/opensolaris/uts/common/zmod/zconf.h head/sys/cddl/contrib/opensolaris/uts/common/zmod/zlib.h head/sys/cddl/contrib/opensolaris/uts/common/zmod/zmod.c head/sys/cddl/contrib/opensolaris/uts/common/zmod/zmod_subr.c head/sys/cddl/contrib/opensolaris/uts/common/zmod/zutil.c head/sys/cddl/contrib/opensolaris/uts/common/zmod/zutil.h head/sys/conf/NOTES head/sys/conf/files head/sys/conf/kern.pre.mk head/sys/conf/kmod.mk head/sys/contrib/zlib/deflate.c head/sys/contrib/zlib/zconf.h head/sys/contrib/zlib/zutil.h head/sys/dev/zlib/ head/sys/dev/zlib/zcalloc.c head/sys/dev/zlib/zcalloc.h head/sys/dev/zlib/zlib_mod.c head/sys/libkern/zlib.c head/sys/modules/crypto/Makefile head/sys/modules/zfs/Makefile head/sys/modules/zlib/Makefile head/sys/opencrypto/cryptodeflate.c head/sys/opencrypto/deflate.h head/sys/sys/zlib.h
A commit references this bug: Author: delphij Date: Sat Aug 3 03:36:19 UTC 2019 New revision: 350554 URL: https://svnweb.freebsd.org/changeset/base/350554 Log: if_mxge: update zlib version 1.0.4 to 1.2.11. PR: 229763 Submitted by: Yoshihiro Ota <ota j email ne jp> Differential Revision: https://reviews.freebsd.org/D20272 Changes: head/share/man/man4/mxge.4 head/sys/conf/files head/sys/dev/mxge/if_mxge.c head/sys/modules/mxge/mxge/Makefile
A commit references this bug: Author: delphij Date: Wed Aug 7 01:41:21 UTC 2019 New revision: 350670 URL: https://svnweb.freebsd.org/changeset/base/350670 Log: Expose zlib's utility functions in Z_SOLO library when building kernel. This allows kernel code to reuse zlib's implementation. PR: 229763 Reviewed by: Yoshihiro Ota <ota j email ne jp> Relnotes: yes Differential Revision: https://reviews.freebsd.org/D21156 Changes: head/sys/conf/files head/sys/conf/kern.pre.mk head/sys/conf/kmod.mk head/sys/contrib/zlib/deflate.c head/sys/contrib/zlib/infback.c head/sys/contrib/zlib/inflate.c head/sys/contrib/zlib/zconf.h head/sys/contrib/zlib/zlib.h head/sys/contrib/zlib/zutil.h head/sys/dev/zlib/zcalloc.c head/sys/dev/zlib/zcalloc.h head/sys/modules/crypto/Makefile head/sys/modules/mxge/mxge/Makefile head/sys/modules/zfs/Makefile head/sys/modules/zlib/Makefile
A commit references this bug: Author: delphij Date: Thu Aug 8 06:27:40 UTC 2019 New revision: 350742 URL: https://svnweb.freebsd.org/changeset/base/350742 Log: Update geom_uzip to use new zlib: - Use new zlib headers; - Removed z_alloc and z_free to use the common sys/dev/zlib version. - Replace z_compressBound with compressBound from zlib. While there, limit LZMA CFLAGS to apply only for g_uzip_lzma.c. PR: 229763 Submitted by: Yoshihiro Ota <ota j email ne jp> (with changes, bugs are mine) Differential Revision: https://reviews.freebsd.org/D20271 Changes: head/sys/geom/uzip/g_uzip_zlib.c head/sys/modules/geom/geom_uzip/Makefile
A commit references this bug: Author: delphij Date: Thu Aug 8 06:30:13 UTC 2019 New revision: 350743 URL: https://svnweb.freebsd.org/changeset/base/350743 Log: Update bxe(4) to use new zlib. PR: 229763 Submitted by: Yoshihiro Ota <ota j email ne jp> Differential Revision: https://reviews.freebsd.org/D21175 Changes: head/sys/dev/bxe/bxe.h
A commit references this bug: Author: delphij Date: Thu Aug 8 07:27:50 UTC 2019 New revision: 350744 URL: https://svnweb.freebsd.org/changeset/base/350744 Log: Convert DDB_CTF to use newer version of ZLIB. PR: 229763 Submitted by: Yoshihiro Ota <ota j email ne jp> Differential Revision: https://reviews.freebsd.org/D21176 Changes: head/sys/kern/kern_ctf.c head/sys/kern/link_elf.c head/sys/kern/link_elf_obj.c
A commit references this bug: Author: delphij Date: Fri Aug 23 07:24:37 UTC 2019 New revision: 351418 URL: https://svnweb.freebsd.org/changeset/base/351418 Log: Convert ng_deflate to use new zlib. This removes the last consumer of the modified zlib originally bundled with Paul's PPP implementation, which will be removed in a follow up commit. PR: 229763 Differential Revision: https://reviews.freebsd.org/D21186 Changes: head/sys/netgraph/ng_deflate.c
A commit references this bug: Author: delphij Date: Sun Aug 25 07:50:45 UTC 2019 New revision: 351477 URL: https://svnweb.freebsd.org/changeset/base/351477 Log: GZIO: Update to use zlib 1.2.11. PR: 229763 Submitted by: Yoshihiro Ota <ota j email ne jp> Differential Revision: https://reviews.freebsd.org/D21408 Changes: head/sys/kern/subr_compressor.c
A commit references this bug: Author: delphij Date: Sun Aug 25 17:13:01 UTC 2019 New revision: 351480 URL: https://svnweb.freebsd.org/changeset/base/351480 Log: Remove zlib 1.0.4 from kernel. PR: 229763 Reviewed by: emaste, Yoshihiro Ota <ota j email ne jp> Differential Revision: https://reviews.freebsd.org/D21375 Changes: head/ObsoleteFiles.inc head/sys/conf/files head/sys/libkern/zlib.c head/sys/modules/zlib/Makefile head/sys/sys/param.h head/sys/sys/zlib.h head/sys/sys/zutil.h
Thanks Yoshihiro Ota for most of the work!
This is fantastic! Thank you Yoshihiro Ota!
Thanks to Xin Li and others who gave us advises and hints, too. Xin took care of all of reviews and commits as well as ng_deflate and ZFS. We can also use compress/uncompress APIs that ease initialization of zlib APIs. That's where I started looking into kernel zlib and drove me for the efforts.
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=254a1740cde85300587fcf4e3ad3b91453fa5d09 commit 254a1740cde85300587fcf4e3ad3b91453fa5d09 Author: Xin LI <delphij@FreeBSD.org> AuthorDate: 2019-08-25 17:13:00 +0000 Commit: Xin LI <delphij@FreeBSD.org> CommitDate: 2022-04-10 04:16:24 +0000 Unify zlib instances into one. This is a cumulative update that consists 14 commits on main by Yoshihiro Ota and myself to stable/12: Remove gzip'ed a.out support. (cherry picked from commit d4565741c6b4b256bd30aea8eb74e8a267cf6b60) Delete unneeded #include <sys/inflate.h> from sys/mips. (cherry picked from commit 880c6c1b06d203fd6b628d313b5b2658fb7135cb) cryptodeflate: Drop z_stream zbuf.state->dummy from SDT probe. (cherry picked from commit a49818787d7cff34b9ea9878e37c1c1ba221f081) Remove kgzip and kgzldr. (cherry picked from commit 5e86bd6073a2fb107318691aaa27b7e19bd45c24) Separate kernel crc32() implementation to its own header (gsb_crc32.h) and rename the source to gsb_crc32.c. (cherry picked from commit f89d2072795407d7c3afff865b988e021c1451a2) Allow Kernel to link in both legacy libkern/zlib and new sys/contrib/zlib. (cherry picked from commit 0ed1d6fb00b8f22c82c3f9054c6dd16eb12469d2) if_mxge: update zlib version 1.0.4 to 1.2.12. (cherry picked from commit 1dbf944a91cfccba2c18fa273f69985cd15a5081) Expose zlib's utility functions in Z_SOLO library when building kernel. This allows kernel code to reuse zlib's implementation. (cherry picked from commit a15cb219c6f2b8ed16179c2fce882a2ff327b753) Update geom_uzip to use new zlib. (cherry picked from commit 2b0cabbdaeac61bdb07909cb0ec995aba36e06b0) Update bxe(4) to use new zlib. (cherry picked from commit 92e9c0608bf161df04d14160084da2d50bcad30e) Convert DDB_CTF to use newer version of ZLIB. (cherry picked from commit 22bbc4b24270ec55c267a5b4ca4530991c3da408) Convert ng_deflate to use new zlib. (cherry picked from commit 34ff55b662d62adea206cb2b77bb064110f68e6d) GZIO: Update to use zlib 1.2.12. (cherry picked from commit 4e8671dd78cc77b31c598e83354fa6722acd4e3f) Remove zlib 1.0.4 from kernel. (cherry picked from commit 21aae72489aba9f1711c51672b3003cf8d54965d) PR: 205822 PR: 229763 ObsoleteFiles.inc | 8 + lib/libufs/Makefile | 3 +- share/man/man4/mxge.4 | 3 +- stand/i386/Makefile | 4 - stand/i386/kgzldr/Makefile (gone) | 20 - stand/i386/kgzldr/Makefile.depend (gone) | 12 - stand/i386/kgzldr/boot.c (gone) | 129 - stand/i386/kgzldr/crt.s (gone) | 83 - stand/i386/kgzldr/kgzldr.h (gone) | 41 - stand/i386/kgzldr/lib.c (gone) | 88 - stand/i386/kgzldr/sio.s (gone) | 44 - stand/i386/kgzldr/start.s (gone) | 45 - stand/libsa/crc32_libkern.c | 2 +- sys/amd64/conf/NOTES | 1 - sys/arm/arm/elf_trampoline.c (gone) | 557 -- .../opensolaris/uts/common/fs/zfs/zfs_ioctl.c | 1 + .../opensolaris/uts/common/zmod/adler32.c (gone) | 149 - .../opensolaris/uts/common/zmod/crc32.h (gone) | 443 -- .../opensolaris/uts/common/zmod/deflate.c (gone) | 1742 ------- .../opensolaris/uts/common/zmod/deflate.h (gone) | 331 -- .../opensolaris/uts/common/zmod/inffast.c (gone) | 320 -- .../opensolaris/uts/common/zmod/inffast.h (gone) | 13 - .../opensolaris/uts/common/zmod/inffixed.h (gone) | 96 - .../opensolaris/uts/common/zmod/inflate.c (gone) | 1395 ----- .../opensolaris/uts/common/zmod/inflate.h (gone) | 117 - .../opensolaris/uts/common/zmod/inftrees.c (gone) | 331 -- .../opensolaris/uts/common/zmod/inftrees.h (gone) | 57 - .../uts/common/zmod/opensolaris_crc32.c (gone) | 428 -- .../opensolaris/uts/common/zmod/trees.c (gone) | 1219 ----- .../opensolaris/uts/common/zmod/zconf.h (gone) | 117 - .../opensolaris/uts/common/zmod/zlib.h (gone) | 1359 ----- .../contrib/opensolaris/uts/common/zmod/zmod.c | 46 +- .../opensolaris/uts/common/zmod/zmod_subr.c (gone) | 66 - .../opensolaris/uts/common/zmod/zutil.c (gone) | 324 -- .../opensolaris/uts/common/zmod/zutil.h (gone) | 274 - sys/conf/NOTES | 2 + sys/conf/files | 41 +- sys/conf/files.amd64 | 1 - sys/conf/files.i386 | 1 - sys/contrib/zlib/deflate.c | 6 +- sys/contrib/zlib/infback.c | 4 +- sys/contrib/zlib/inflate.c | 4 +- sys/contrib/zlib/zconf.h | 15 +- sys/contrib/zlib/zlib.h | 5 +- sys/contrib/zlib/zutil.h | 2 +- sys/dev/bxe/bxe.h | 2 +- sys/dev/iscsi/icl_soft.c | 1 + sys/dev/iscsi_initiator/isc_subr.c | 1 + sys/dev/liquidio/lio_bsd.h | 1 + sys/dev/mxge/if_mxge.c | 23 +- sys/dev/usb/net/if_cdce.c | 1 + sys/dev/usb/net/if_cdceem.c | 1 + sys/dev/zlib/zcalloc.c (new) | 39 + sys/dev/zlib/zcalloc.h (new) | 14 + sys/dev/zlib/zlib_mod.c (new) | 53 + sys/fs/ext2fs/ext2_csum.c | 1 + sys/fs/nandfs/nandfs_segment.c | 1 + sys/fs/nandfs/nandfs_subr.c | 1 + sys/fs/nandfs/nandfs_vfsops.c | 1 + sys/geom/part/g_part_bsd64.c | 1 + sys/geom/part/g_part_gpt.c | 1 + sys/geom/raid/md_ddf.c | 1 + sys/geom/uzip/g_uzip_zlib.c | 35 +- sys/i386/conf/NOTES | 1 - sys/kern/imgact_gzip.c (gone) | 394 -- sys/kern/kern_ctf.c | 18 +- sys/kern/link_elf.c | 4 - sys/kern/link_elf_obj.c | 2 +- sys/kern/subr_compressor.c | 9 +- sys/kern/subr_inflate.c (gone) | 1084 ---- sys/libkern/{crc32.c => gsb_crc32.c} | 1 + sys/libkern/x86/crc32_sse42.c | 9 +- sys/libkern/zlib.c (gone) | 5414 -------------------- sys/mips/mips/elf_trampoline.c | 1 - sys/modules/geom/geom_uzip/Makefile | 2 +- sys/modules/zfs/Makefile | 9 - sys/modules/zlib/Makefile | 19 +- sys/netgraph/ng_deflate.c | 97 +- sys/netinet/libalias/alias_sctp.c | 1 + sys/netinet/sctp_crc32.c | 1 + sys/netpfil/pf/pf.c | 1 + sys/opencrypto/cryptodeflate.c | 97 +- sys/opencrypto/deflate.h | 4 - sys/sys/gsb_crc32.h (new) | 47 + sys/sys/inflate.h (gone) | 53 - sys/sys/libkern.h | 33 - sys/sys/zlib.h (gone) | 1020 ---- sys/sys/zutil.h (gone) | 228 - sys/ufs/ffs/ffs_alloc.c | 1 + sys/ufs/ffs/ffs_snapshot.c | 1 + sys/ufs/ffs/ffs_subr.c | 1 + sys/ufs/ffs/ffs_vfsops.c | 1 + targets/pseudo/userland/Makefile.depend | 1 - targets/pseudo/userland/misc/Makefile.depend | 2 +- tests/sys/kern/Makefile | 1 - tests/sys/kern/libkern_crc32.c | 7 +- usr.sbin/Makefile.amd64 | 1 - usr.sbin/Makefile.i386 | 1 - usr.sbin/kgzip/Makefile (gone) | 9 - usr.sbin/kgzip/Makefile.depend (gone) | 15 - usr.sbin/kgzip/aouthdr.c (gone) | 81 - usr.sbin/kgzip/aouthdr.h (gone) | 61 - usr.sbin/kgzip/elfhdr.c (gone) | 166 - usr.sbin/kgzip/elfhdr.h (gone) | 86 - usr.sbin/kgzip/kgz.h (gone) | 59 - usr.sbin/kgzip/kgzcmp.c (gone) | 239 - usr.sbin/kgzip/kgzip.8 (gone) | 156 - usr.sbin/kgzip/kgzip.c (gone) | 178 - usr.sbin/kgzip/kgzip.h (gone) | 53 - usr.sbin/kgzip/kgzld.c (gone) | 103 - usr.sbin/kgzip/xio.c (gone) | 123 - 111 files changed, 423 insertions(+), 19599 deletions(-)