Bug 247211 - New port: sysutils/nss_ndb - a fast BDB based nsswitch backend for big passwd & group's
Summary: New port: sysutils/nss_ndb - a fast BDB based nsswitch backend for big passwd...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Kurt Jaeger
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-12 19:26 UTC by Peter Eriksson
Modified: 2020-06-13 20:35 UTC (History)
2 users (show)

See Also:


Attachments
Patch/diff for a new port sysutils/nss_ndb (2.55 KB, patch)
2020-06-12 19:26 UTC, Peter Eriksson
no flags Details | Diff
Updated diff that fixes the portlint errors (2.82 KB, patch)
2020-06-12 20:28 UTC, Peter Eriksson
no flags Details | Diff
v3 of the diff - fixes USES localbase & USE_LDCONFIG (2.71 KB, patch)
2020-06-13 08:44 UTC, Peter Eriksson
no flags Details | Diff
Attempt #4 - new version of the diff... (3.10 KB, patch)
2020-06-13 19:40 UTC, Peter Eriksson
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Eriksson 2020-06-12 19:26:58 UTC
Created attachment 215492 [details]
Patch/diff for a new port sysutils/nss_ndb

A new port: 

nss_ndb is a nsswitch backend that uses BerkeleyDB (by default the old builtin version in libc, but can optionally use v6, v5 or v4.8) that is thread-safe and optimized for speed and handling of large passwd & group maps. Also supports incremental updates of the databases.

It has been in use on our systems for a couple of years handling ~140000 passwd entries and thousands of groups (talking directly to AD via Samba winbind causes slowdowns so we "sync" the AD contents into the nss_ndb databases).

- Peter
Comment 1 Kurt Jaeger freebsd_committer freebsd_triage 2020-06-12 19:59:58 UTC
Thanks for the patch. Can you run

portlint -AC

in the new ports directory' ?

When I do this, it gives me many warnings...
Comment 2 Peter Eriksson 2020-06-12 20:14:16 UTC
Argh. I must have missed that on that port. On it... Brb :-)
Comment 3 Peter Eriksson 2020-06-12 20:28:59 UTC
Created attachment 215494 [details]
Updated diff that fixes the portlint errors

Updated version of the diff that fixes the portlint complaints.
Comment 5 Peter Eriksson 2020-06-13 08:44:19 UTC
Created attachment 215514 [details]
v3 of the diff - fixes USES localbase & USE_LDCONFIG

Ah, I was looking for the right way to detect the location of the db libraries (USE=localbase and USE_LDCONFIG) but I must have missed it in the documentation (it's probably obvious when you have ported a lot of stuff but I'm new to this :-). 

Ok, updated version of the diff that fixes those issues now uploaded.
Comment 6 Kurt Jaeger freebsd_committer freebsd_triage 2020-06-13 09:49:24 UTC
Thanks for the updated patch. Did you test this in poudriere ?

I tested it on cur-amd64, cur-i386, 12.1-amd64, 11.4-i386 and 11.3-amd64
and it failed with different staging errors:

113, 11.4, 12.1:
/usr/bin/install -c -o root -g wheel -m 0755 nss_ndb.so.1.0.22 /wrkdirs/usr/ports/sysutils/nss_ndb/work/stage/usr/local/lib
install: /wrkdirs/usr/ports/sysutils/nss_ndb/work/stage/usr/local/lib/nss_ndb.so.1.0.22: chown/chgrp: Operation not permitted

cur/cur-i:
===> Checking for items in STAGEDIR missing from pkg-plist
Error: Orphaned: /ndbsync
Error: Orphaned: bin/nsstest
Error: Orphaned: %%ETCDIR%%.conf
Error: Orphaned: @dir /var/db/nss_ndb

I know setting up poudriere is a hurdle, but can you give it a try ?
Comment 7 Kurt Jaeger freebsd_committer freebsd_triage 2020-06-13 10:14:07 UTC
(In reply to Kurt Jaeger from comment #6)
And, I guess, having some file as /ndbsync does not look right.
Comment 8 Peter Eriksson 2020-06-13 19:40:54 UTC
Created attachment 215524 [details]
Attempt #4 - new version of the diff...

Right... Ok, I've fixed those problems now. And tested it with "portlint -AC" and "port test" too. And my own tests. 

Port builds/tested on FreeBSD 11.3-RELEASE-p10 and 12.1-RELEASE-p6. amd64 both.

Also updated the base package on github with a couple of fixes that was needed while I was at it (some manual pages are there now for example :-)

I'll look into Poudriere for the future...
Comment 9 Kurt Jaeger freebsd_committer freebsd_triage 2020-06-13 20:35:09 UTC
Committed, thanks!
Comment 10 commit-hook freebsd_committer freebsd_triage 2020-06-13 20:35:40 UTC
A commit references this bug:

Author: pi
Date: Sat Jun 13 20:34:56 UTC 2020
New revision: 538698
URL: https://svnweb.freebsd.org/changeset/ports/538698

Log:
  New port: sysutils/nss_ndb BerkeleyDB-based nsswitch backend for passwd & groups

  This project consists of a NSS module (nss_ndb.so.1) and a CLI
  utility (makendb) that enables big passwd & group files to be
  handled efficiently via BTree BerkeleyDB databases.

  To activate and use this you must (after installation):

  1. Manually add a symbolic link:

     /usr/lib/nss_ndb.so.1 -> /path/to/lib/nss_ndb.so.1.<version>

  2. Populate the databases in /var/db/nss_ndb via "makendb" (or some
     other tool).

  3. Activate it in /etc/nsswitch.conf:

     passwd: files ndb
     group:  files ndb

  WWW: https://github.com/ptrrkssn/nss_ndb

  PR:		247211
  Submitted by:	Peter Eriksson <pen@lysator.liu.se>

Changes:
  head/sysutils/Makefile
  head/sysutils/nss_ndb/
  head/sysutils/nss_ndb/Makefile
  head/sysutils/nss_ndb/distinfo
  head/sysutils/nss_ndb/pkg-descr