Bug 229763 - Consolidate multiple zlib instances in kernel
Summary: Consolidate multiple zlib instances in kernel
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Many People
Assignee: Xin LI
URL:
Keywords: patch
Depends on:
Blocks:
 
Reported: 2018-07-13 23:10 UTC by Xin LI
Modified: 2019-08-25 22:10 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Xin LI freebsd_committer 2018-07-13 23:10:36 UTC
Currently there are multiple zlib variants in kernel:

sys/cddl/contrib/opensolaris/uts/common/zmod/zmod.c
sys/kern/subr_inflate.c
sys/libkern/zlib.c

There should be only one copy of zlib, ideally share the same copy of contrib/zlib.
Comment 1 Xin LI freebsd_committer 2018-07-13 23:11:17 UTC
Earlier work to consolidate zlib instances:

https://svnweb.freebsd.org/base?view=revision&revision=199954
Comment 2 ota 2019-03-26 02:49:34 UTC
That SVN revision is too old and won't easily apply to the latest.

I took a different approach and created https://reviews.freebsd.org/D19706
Comment 3 Ed Maste freebsd_committer 2019-05-21 19:14:19 UTC
Assign to delphij@ for tracking, this is in progress.
Comment 4 commit-hook freebsd_committer 2019-05-23 05:18:19 UTC
A commit references this bug:

Author: delphij
Date: Thu May 23 05:17:19 UTC 2019
New revision: 348148
URL: https://svnweb.freebsd.org/changeset/base/348148

Log:
  Delete unneeded #include <sys/inflate.h> from sys/mips.

  PR:		229763
  Submitted by:	Yoshihiro Ota <ota at j.email.ne.jp>
  Reviewed by:	imp
  Differential Revision:	https://reviews.freebsd.org/D20190

Changes:
  head/sys/mips/mips/elf_trampoline.c
Comment 5 commit-hook freebsd_committer 2019-05-24 02:45:13 UTC
A commit references this bug:

Author: delphij
Date: Fri May 24 02:44:16 UTC 2019
New revision: 348222
URL: https://svnweb.freebsd.org/changeset/base/348222

Log:
  cryptodeflate: Drop z_stream zbuf.state->dummy from SDT probe.

  For older versions of zlib, dummy was a workaround for compilers that do not
  handle opaque type definition well; on FreeBSD, it's representing a value
  that is not really useful for monitoring purposes, and the field would be gone
  in newer zlib versions.

  PR:		229763
  Submitted by:	Yoshihiro Ota <ota at j.email.ne.jp>
  Differential Revision:	https://reviews.freebsd.org/D20222

Changes:
  head/sys/opencrypto/cryptodeflate.c
Comment 6 commit-hook freebsd_committer 2019-05-24 05:34:35 UTC
A commit references this bug:

Author: delphij
Date: Fri May 24 05:34:23 UTC 2019
New revision: 348225
URL: https://svnweb.freebsd.org/changeset/base/348225

Log:
  Remove kgzip and kgzldr.

  PR:		183666, 229763
  Submitted by:	Yoshihiro Ota <ota at j.email.ne.jp>
  Differential Revision:	https://reviews.freebsd.org/D20248

Changes:
  head/ObsoleteFiles.inc
  head/stand/i386/Makefile
  head/stand/i386/kgzldr/
  head/targets/pseudo/userland/Makefile.depend
  head/targets/pseudo/userland/misc/Makefile.depend
  head/usr.sbin/Makefile.amd64
  head/usr.sbin/Makefile.i386
  head/usr.sbin/kgzip/
Comment 7 commit-hook freebsd_committer 2019-06-17 21:02:04 UTC
A commit references this bug:

Author: delphij
Date: Mon Jun 17 19:49:15 UTC 2019
New revision: 349151
URL: https://svnweb.freebsd.org/changeset/base/349151

Log:
  Separate kernel crc32() implementation to its own header (gsb_crc32.h) and
  rename the source to gsb_crc32.c.

  This is a prerequisite of unifying kernel zlib instances.

  PR:		229763
  Submitted by:	Yoshihiro Ota <ota at j.email.ne.jp>
  Differential Revision:	https://reviews.freebsd.org/D20193

Changes:
  head/lib/libufs/Makefile
  head/stand/libsa/crc32_libkern.c
  head/sys/conf/files
  head/sys/dev/iscsi/icl_soft.c
  head/sys/dev/iscsi_initiator/isc_subr.c
  head/sys/dev/liquidio/lio_bsd.h
  head/sys/dev/usb/net/if_cdce.c
  head/sys/fs/ext2fs/ext2_csum.c
  head/sys/fs/nandfs/nandfs_segment.c
  head/sys/fs/nandfs/nandfs_subr.c
  head/sys/fs/nandfs/nandfs_vfsops.c
  head/sys/geom/part/g_part_bsd64.c
  head/sys/geom/part/g_part_gpt.c
  head/sys/geom/raid/md_ddf.c
  head/sys/kern/subr_compressor.c
  head/sys/libkern/crc32.c
  head/sys/libkern/gsb_crc32.c
  head/sys/libkern/x86/crc32_sse42.c
  head/sys/netinet/libalias/alias_sctp.c
  head/sys/netinet/sctp_crc32.c
  head/sys/netpfil/pf/pf.c
  head/sys/sys/gsb_crc32.h
  head/sys/sys/libkern.h
  head/sys/sys/param.h
  head/sys/ufs/ffs/ffs_alloc.c
  head/sys/ufs/ffs/ffs_snapshot.c
  head/sys/ufs/ffs/ffs_subr.c
  head/sys/ufs/ffs/ffs_vfsops.c
  head/tests/sys/kern/Makefile
  head/tests/sys/kern/libkern_crc32.c
Comment 8 commit-hook freebsd_committer 2019-08-01 06:35:54 UTC
A commit references this bug:

Author: delphij
Date: Thu Aug  1 06:35:36 UTC 2019
New revision: 350496
URL: https://svnweb.freebsd.org/changeset/base/350496

Log:
  Allow Kernel to link in both legacy libkern/zlib and new sys/contrib/zlib,
  with an eventual goal to convert all legacl zlib callers to the new zlib
  version:

   * Move generic zlib shims that are not specific to zlib 1.0.4 to
     sys/dev/zlib.
   * Connect new zlib (1.2.11) to the zlib kernel module, currently built
     with Z_SOLO.
   * Prefix the legacy zlib (1.0.4) with 'zlib104_' namespace.
   * Convert sys/opencrypto/cryptodeflate.c to use new zlib.
   * Remove bundled zlib 1.2.3 from ZFS and adapt it to new zlib and make
     it depend on the zlib module.
   * Fix Z_SOLO build of new zlib.

  PR:		229763
  Submitted by:	Yoshihiro Ota <ota j email ne jp>
  Reviewed by:	markm (sys/dev/zlib/zlib_kmod.c)
  Relnotes:	yes
  Differential Revision:	https://reviews.freebsd.org/D19706

Changes:
  head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/adler32.c
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/crc32.h
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/deflate.c
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/deflate.h
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/inffast.c
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/inffast.h
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/inffixed.h
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/inflate.c
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/inflate.h
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/inftrees.c
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/inftrees.h
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/opensolaris_crc32.c
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/trees.c
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/zconf.h
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/zlib.h
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/zmod.c
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/zmod_subr.c
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/zutil.c
  head/sys/cddl/contrib/opensolaris/uts/common/zmod/zutil.h
  head/sys/conf/NOTES
  head/sys/conf/files
  head/sys/conf/kern.pre.mk
  head/sys/conf/kmod.mk
  head/sys/contrib/zlib/deflate.c
  head/sys/contrib/zlib/zconf.h
  head/sys/contrib/zlib/zutil.h
  head/sys/dev/zlib/
  head/sys/dev/zlib/zcalloc.c
  head/sys/dev/zlib/zcalloc.h
  head/sys/dev/zlib/zlib_mod.c
  head/sys/libkern/zlib.c
  head/sys/modules/crypto/Makefile
  head/sys/modules/zfs/Makefile
  head/sys/modules/zlib/Makefile
  head/sys/opencrypto/cryptodeflate.c
  head/sys/opencrypto/deflate.h
  head/sys/sys/zlib.h
Comment 9 commit-hook freebsd_committer 2019-08-03 03:36:35 UTC
A commit references this bug:

Author: delphij
Date: Sat Aug  3 03:36:19 UTC 2019
New revision: 350554
URL: https://svnweb.freebsd.org/changeset/base/350554

Log:
  if_mxge: update zlib version 1.0.4 to 1.2.11.

  PR:		229763
  Submitted by:	Yoshihiro Ota <ota j email ne jp>
  Differential Revision:	https://reviews.freebsd.org/D20272

Changes:
  head/share/man/man4/mxge.4
  head/sys/conf/files
  head/sys/dev/mxge/if_mxge.c
  head/sys/modules/mxge/mxge/Makefile
Comment 10 commit-hook freebsd_committer 2019-08-07 01:41:31 UTC
A commit references this bug:

Author: delphij
Date: Wed Aug  7 01:41:21 UTC 2019
New revision: 350670
URL: https://svnweb.freebsd.org/changeset/base/350670

Log:
  Expose zlib's utility functions in Z_SOLO library when building kernel.
  This allows kernel code to reuse zlib's implementation.

  PR:		229763
  Reviewed by:	Yoshihiro Ota <ota j email ne jp>
  Relnotes:	yes
  Differential Revision:	https://reviews.freebsd.org/D21156

Changes:
  head/sys/conf/files
  head/sys/conf/kern.pre.mk
  head/sys/conf/kmod.mk
  head/sys/contrib/zlib/deflate.c
  head/sys/contrib/zlib/infback.c
  head/sys/contrib/zlib/inflate.c
  head/sys/contrib/zlib/zconf.h
  head/sys/contrib/zlib/zlib.h
  head/sys/contrib/zlib/zutil.h
  head/sys/dev/zlib/zcalloc.c
  head/sys/dev/zlib/zcalloc.h
  head/sys/modules/crypto/Makefile
  head/sys/modules/mxge/mxge/Makefile
  head/sys/modules/zfs/Makefile
  head/sys/modules/zlib/Makefile
Comment 11 commit-hook freebsd_committer 2019-08-08 06:28:04 UTC
A commit references this bug:

Author: delphij
Date: Thu Aug  8 06:27:40 UTC 2019
New revision: 350742
URL: https://svnweb.freebsd.org/changeset/base/350742

Log:
  Update geom_uzip to use new zlib:

   - Use new zlib headers;
   - Removed z_alloc and z_free to use the common sys/dev/zlib version.
   - Replace z_compressBound with compressBound from zlib.

  While there, limit LZMA CFLAGS to apply only for g_uzip_lzma.c.

  PR:		229763
  Submitted by:	Yoshihiro Ota <ota j email ne jp> (with changes,
  		bugs are mine)
  Differential Revision:	https://reviews.freebsd.org/D20271

Changes:
  head/sys/geom/uzip/g_uzip_zlib.c
  head/sys/modules/geom/geom_uzip/Makefile
Comment 12 commit-hook freebsd_committer 2019-08-08 06:31:07 UTC
A commit references this bug:

Author: delphij
Date: Thu Aug  8 06:30:13 UTC 2019
New revision: 350743
URL: https://svnweb.freebsd.org/changeset/base/350743

Log:
  Update bxe(4) to use new zlib.

  PR:		229763
  Submitted by:	Yoshihiro Ota <ota j email ne jp>
  Differential Revision:	https://reviews.freebsd.org/D21175

Changes:
  head/sys/dev/bxe/bxe.h
Comment 13 commit-hook freebsd_committer 2019-08-08 07:28:13 UTC
A commit references this bug:

Author: delphij
Date: Thu Aug  8 07:27:50 UTC 2019
New revision: 350744
URL: https://svnweb.freebsd.org/changeset/base/350744

Log:
  Convert DDB_CTF to use newer version of ZLIB.

  PR:		229763
  Submitted by:	Yoshihiro Ota <ota j email ne jp>
  Differential Revision:	https://reviews.freebsd.org/D21176

Changes:
  head/sys/kern/kern_ctf.c
  head/sys/kern/link_elf.c
  head/sys/kern/link_elf_obj.c
Comment 14 commit-hook freebsd_committer 2019-08-23 07:24:48 UTC
A commit references this bug:

Author: delphij
Date: Fri Aug 23 07:24:37 UTC 2019
New revision: 351418
URL: https://svnweb.freebsd.org/changeset/base/351418

Log:
  Convert ng_deflate to use new zlib.

  This removes the last consumer of the modified zlib originally
  bundled with Paul's PPP implementation, which will be removed
  in a follow up commit.

  PR:			229763
  Differential Revision:	https://reviews.freebsd.org/D21186

Changes:
  head/sys/netgraph/ng_deflate.c
Comment 15 commit-hook freebsd_committer 2019-08-25 07:50:58 UTC
A commit references this bug:

Author: delphij
Date: Sun Aug 25 07:50:45 UTC 2019
New revision: 351477
URL: https://svnweb.freebsd.org/changeset/base/351477

Log:
  GZIO: Update to use zlib 1.2.11.

  PR:		229763
  Submitted by:	Yoshihiro Ota <ota j email ne jp>
  Differential Revision:	https://reviews.freebsd.org/D21408

Changes:
  head/sys/kern/subr_compressor.c
Comment 16 commit-hook freebsd_committer 2019-08-25 17:13:10 UTC
A commit references this bug:

Author: delphij
Date: Sun Aug 25 17:13:01 UTC 2019
New revision: 351480
URL: https://svnweb.freebsd.org/changeset/base/351480

Log:
  Remove zlib 1.0.4 from kernel.

  PR:		229763
  Reviewed by:	emaste, Yoshihiro Ota <ota j email ne jp>
  Differential Revision:	https://reviews.freebsd.org/D21375

Changes:
  head/ObsoleteFiles.inc
  head/sys/conf/files
  head/sys/libkern/zlib.c
  head/sys/modules/zlib/Makefile
  head/sys/sys/param.h
  head/sys/sys/zlib.h
  head/sys/sys/zutil.h
Comment 17 Xin LI freebsd_committer 2019-08-25 17:18:13 UTC
Thanks Yoshihiro Ota for most of the work!
Comment 18 Conrad Meyer freebsd_committer 2019-08-25 17:25:24 UTC
This is fantastic!  Thank you Yoshihiro Ota!
Comment 19 ota 2019-08-25 22:10:19 UTC
Thanks to Xin Li and others who gave us advises and hints, too.
Xin took care of all of reviews and commits as well as ng_deflate and ZFS.

We can also use compress/uncompress APIs that ease initialization of zlib APIs.
That's where I started looking into kernel zlib and drove me for the efforts.