There are some FreeBSD boxes which have really huge master.passwd files since we have no nss under 4.x and, thus, all users should be listed there for a lot of programs. The host is described as FIRST above is the mailboxes and homepages storage for the all users of some ISP in Moscow, Russia. It holds above the 10000 lines in the master.passwd. There is an option for pwd_mkdb which allows to set the size of cache when database is opened (-s <cache size in Ms>). That helps sufficiently increase the speed of database rebuilding. But there no way to use this option with the pw_mkdb() function, shipped with vipw in 4.x and moved to libutil in CURRENT. That function execs pwd_mkdb with options -p -d <dir>, and optionally with -u <user>. Vipw (and maybe some other programs in -CURRENT) just uses the call to pw_mkdb(). So there is no way for vipw to set somehow the cache size to be used in pwd_mkdb while it can be very useful. Some data for the real host are given below ("uname -a" output and the dmesg part about the processor and memory see above, marked as FIRST). I measured the time necessary for the pwd_mkdb to rebuild master.passwd database with the same arguments as used in pw_mkdb() with the "-s" option added. Here are results: cache sized total time to run passed with -s 1 1m1.434s 2 0m32.604s 4 0m3.550s 8 0m2.792s 12 0m2.749s 16 0m2.528s bash-2.05b# wc -l master.passwd 10021 master.passwd Default value of the cache size is 2M. As you can see it can cause very slow execution of pwd_mkdb. The next one - data for the another computer (data are given above under the SECOND label), here master.passwd are generated artificially: cache sized total time to run passed with -s 2 2m54.342s 4 2m18.354s 8 0m39.987s 12 0m10.734s 16 0m7.535s 24 0m7.469s 32 0m7.728s bash-2.05b# wc -l master.passwd 37038 master.passwd Now, even 8M is not enough. Fix: I suggest to pass the value of the cache size through the environment variable. I called it PWDB_CACHE. By default the behaviour of pkd_mkdb and all programs those can inherit the problem does not change. But if system administrator needs to increase the speed of the users db rebuilding it just can set PWDB_CACHE to some value greater than 2 (2M is build-in default) and enjoy. As in command-line the unit is megabyte. Here is the patch: How-To-Repeat: Create a really big (10000-20000-... records) master.passwd file and run #pwd_mkdb -p -s N -d /etc master.passwd where N is 2 to 16.
On Sat, 19 Apr 2003, Alex Semenyaka wrote: > Hello, > > could somebody to comment PR bin/51148? It is suggestion how to pass > a value of cache size to pwd_mkdb when we are doing vipw or such. > It can give a greate speed-up when master.passwd is really big (and > sometimes it is). Appropriate cache size can make process 10 to 100 > or more times faster. I gave the results of measurements in that > problem report. Having been in a "mondo huge master.passwd file" situation myself, my comment is that people in this situation should probably not be relying on tools like vipw to manage their stuff. However, my feelings aren't strong enough to prompt me to close your PR, so good luck. :) Doug -- This .signature sanitized for your protection
On Sun, 20 Apr 2003, Doug Barton wrote: > On Sat, 19 Apr 2003, Alex Semenyaka wrote: > > > Hello, > > > > could somebody to comment PR bin/51148? It is suggestion how to pass > > a value of cache size to pwd_mkdb when we are doing vipw or such. > > It can give a greate speed-up when master.passwd is really big (and > > sometimes it is). Appropriate cache size can make process 10 to 100 > > or more times faster. I gave the results of measurements in that > > problem report. > > Having been in a "mondo huge master.passwd file" situation myself, my > comment is that people in this situation should probably not be relying on > tools like vipw to manage their stuff. However, my feelings aren't strong > enough to prompt me to close your PR, so good luck. :) Are your feelings strong enough to commit bin/38676 so that people can use pw(8) safely and concurrently on mondo huge master.passwd files? :) bin/23501 was supposed to have fixed the problem, but it doesn't appear to have been committed before it was closed. bin/38676 is a revised/improved version of the original patch in bin/23501. Geoff
Hi, On Sun, Apr 20, 2003 at 02:11:34PM -0700, Doug Barton wrote: > Having been in a "mondo huge master.passwd file" situation myself, my > comment is that people in this situation should probably not be relying on > tools like vipw to manage their stuff. Actually, there is a such corporative policy in the ISP I've taken that huge master.passwd for the test. The main ideas: 1) no automatic changes in the master.passwd 2) time to time visual check and fix of users' data in master.passwd (heh, column from 'master.passwd's). May be good, may be bad, but it exists. > However, my feelings aren't strong > enough to prompt me to close your PR, so good luck. :) Thanks :) SY, Alex
For bugs matching the following criteria: Status: In Progress Changed: (is less than) 2014-06-01 Reset to default assignee and clear in-progress tags. Mail being skipped
Keyword: patch or patch-ready – in lieu of summary line prefix: [patch] * bulk change for the keyword * summary lines may be edited manually (not in bulk). Keyword descriptions and search interface: <https://bugs.freebsd.org/bugzilla/describekeywords.cgi>