Bug 248111 - databases/couchdb3 Fails to Start
Summary: databases/couchdb3 Fails to Start
Status: Closed Not Accepted
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Dave Cottlehuber
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-07-19 18:46 UTC by Mike Keen
Modified: 2020-08-01 15:38 UTC (History)
0 users

See Also:
bugzilla: maintainer-feedback? (dch)


Attachments
erl_crash.dump (96.84 KB, text/plain)
2020-07-19 18:46 UTC, Mike Keen
no flags Details
erl_crash.dump (96.84 KB, text/plain)
2020-07-19 22:03 UTC, Mike Keen
no flags Details
erl_crash.dump2 (75.62 KB, text/plain)
2020-07-19 22:03 UTC, Mike Keen
no flags Details
erl_crash.dump3 (85.67 KB, text/plain)
2020-07-19 22:04 UTC, Mike Keen
no flags Details
local.ini (445 bytes, text/plain)
2020-07-19 22:04 UTC, Mike Keen
no flags Details
pkg info (8.95 KB, text/plain)
2020-07-19 22:05 UTC, Mike Keen
no flags Details
uname -a (123 bytes, text/plain)
2020-07-19 22:05 UTC, Mike Keen
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mike Keen 2020-07-19 18:46:21 UTC
Created attachment 216580 [details]
erl_crash.dump

FreeBSD 12.1 x64 zfs

Fresh install from pkg as well as compiled from ports results in the same issue... Upon starting the service, it immediately crashes. This happens whether or not an admin account is created and default interface and port are specified (as is required by CouchDB 3)

The Erlang crash dump is attached... let me know if more info is needed.
Comment 1 Dave Cottlehuber freebsd_committer 2020-07-19 21:03:18 UTC
thanks for the report. Some more detail is needed, I can't reproduce it
here yet.

Does this actually affect many people? So far there is 1 bug report. I'm
running it here on a variety of installs, albeit none from pkg.freebsd.org.

This looks a bit like it bombs out in the scheduler, on a physical (virtual?)
2 core system, very very early on. How were these packages installed, are you
certain that both erlang-runtime22 & couchdb3 are *both* updated from latest
pkg repo?

if you start a plain erlang, does that blow up too?

export PATH=/usr/local/lib/erlang22/bin:$PATH
erl -init_debug -s crypto -eval 'io:format("~w~n",[crypto:strong_rand_bytes(16)]).'

if you run plain erlang, with the same parameters from your vm.args, does
that start or not? include the -init_debug call from above.

if you're using public packages, does this bomb out on a fresh install too?

i.e. a single node install, with no existing config nor db?

if you add `-init_debug` to your vm.args, do we see anything more useful
on startup & crash?

can you attach output `pkg info`, uname -a, your local.ini (without the
admin section and any secrets like [couch_httpd_auth])

couchdb3-3.1.0_1               JSON document database with HTTP API and scalable multi-master sync
erlang-runtime22-22.3.4.3      Functional programming language from Ericsson
icu-67.1,1                     International Components for Unicode (from IBM)
oniguruma-6.9.5.r1_1           Regular expressions library compatible with POSIX/GNU/Perl
pcre-8.44                      Perl Compatible Regular Expressions library
pcre2-10.35                    Perl Compatible Regular Expressions library, version 2
spidermonkey60-60.9.0_4        Standalone JavaScript based from Mozilla 60-esr

FWIW running this in a vanilla 12.1p5 amd64 jail, following instructions in
`pkg info -D couchdb3` produces:

# tee -a /usr/local/etc/couchdb3/local.ini << DEFAULTS

[admins]
admin = passwd
[log]
level = debug
include_sasl = true
writer = stderr
DEFAULTS

# service couchdb3 onestart
# ps auxwwd

USER      PID %CPU %MEM     VSZ   RSS TT  STAT STARTED    TIME COMMAND
couchdb 55903 29.9  0.0 1196772 45184  -  SJ   20:50   0:01.87 /usr/local/lib/erlang22/erts-10.7.2.1/bin/beam.smp -A 16 -Bd -- -root /usr/local/lib/erlang22 -progname erl -- -home /var/run/couchdb -- -sasl releases_dir "couchdb3/releases" -boot /usr/local/libexec/couchdb3/releases/3.1.0/couchdb -boot_var RELTOOL_EXT_LIB /usr/local/libexec/couchdb3/lib -name couchdb@127.0.0.1 -setcookie monster -kernel error_logger silent -sasl sasl_error_logger false -noshell -noinput -noshell -noinput -smp enable -ssl session_lifetime 300 -- -couch_ini /usr/local/libexec/couchdb3/etc/default.ini /usr/local/etc/couchdb3/local.ini -- /usr/local/etc/couchdb3/local.ini --
couchdb 72632  0.3  0.0   10732  2208  -  SsJ  20:50   0:00.00 - erl_child_setup 65000
couchdb 16311  0.1  0.0   11252  2676  -  SJ   20:50   0:00.00 /usr/local/lib/erlang22/erts-10.7.2.1/bin/epmd -daemon
root    41401  0.0  0.0   13536  4904  -  SsJ  20:40   0:00.35 tmux: server (/tmp//tmux-0/default) (tmux)
root    64458  0.0  0.0   13596  4072  3  IsJ  20:40   0:00.03 - -csh (csh)
root    81362  0.0  0.0   12124  3180  3  SJ   20:45   0:00.02 `-- sh
root       59  0.0  0.0   11588  2924  3  R+J  20:50   0:00.00   `-- ps auxwwd
root       58  0.0  0.0   13044  4144  2  I+J  20:40   0:00.02 tmux: client (/tmp//tmux-0/default) (tmux)

# curl -X PUT http://admin:passwd@127.0.0.1:5984/_users
{"ok":true}
# curl -X PUT http://admin:passwd@127.0.0.1:5984/_replicator
{"ok":true}
# curl -X PUT http://admin:passwd@127.0.0.1:5984/_global_changes
{"ok":true}


Is there anything unusual about this box?
Comment 2 Mike Keen 2020-07-19 22:03:02 UTC
Created attachment 216586 [details]
erl_crash.dump

Fresh install, start attempted with service couchdb3 onestart
Comment 3 Mike Keen 2020-07-19 22:03:43 UTC
Created attachment 216587 [details]
erl_crash.dump2

/usr/local/lib/erlang22/bin/erl -init_debug -s crypto -eval 'io:format("~w~n",[crypto:strong_rand_bytes(16)]).' -args_file /usr/local/etc/couchdb3/vm.args
Comment 4 Mike Keen 2020-07-19 22:04:15 UTC
Created attachment 216588 [details]
erl_crash.dump3

/usr/local/lib/erlang22/bin/erl -init_debug -args_file /usr/local/etc/couchdb3/vm.args
Comment 5 Mike Keen 2020-07-19 22:04:50 UTC
Created attachment 216589 [details]
local.ini
Comment 6 Mike Keen 2020-07-19 22:05:08 UTC
Created attachment 216590 [details]
pkg info
Comment 7 Mike Keen 2020-07-19 22:05:29 UTC
Created attachment 216591 [details]
uname -a
Comment 8 Mike Keen 2020-07-19 22:06:08 UTC
(In reply to Dave Cottlehuber from comment #1)

Plain Erlang does not blow up.. output below

root@couchdb-a:~ # /usr/local/lib/erlang22/bin/erl -init_debug -s crypto -eval 'io:format("~w~n",[crypto:strong_rand_bytes(16)]).'
{progress,preloaded}
{progress,kernel_load_completed}
{progress,modules_loaded}
{start,heart}
{start,logger}
{start,application_controller}
{progress,init_kernel_started}
{apply,{application,load,[{application,stdlib,[{description,"ERTS  CXC 138 10"},{vsn,"3.12.1"},{id,[]},{modules,[array,base64,beam_lib,binary,c,calendar,dets,dets_server,dets_sup,dets_utils,dets_v9,dict,digraph,digraph_utils,edlin,edlin_expand,epp,eval_bits,erl_abstract_code,erl_anno,erl_bits,erl_compile,erl_error,erl_eval,erl_expand_records,erl_internal,erl_lint,erl_parse,erl_posix_msg,erl_pp,erl_scan,erl_tar,error_logger_file_h,error_logger_tty_h,escript,ets,file_sorter,filelib,filename,gb_trees,gb_sets,gen,gen_event,gen_fsm,gen_server,gen_statem,io,io_lib,io_lib_format,io_lib_fread,io_lib_pretty,lists,log_mf_h,maps,math,ms_transform,orddict,ordsets,otp_internal,pool,proc_lib,proplists,qlc,qlc_pt,queue,rand,random,re,sets,shell,shell_default,slave,sofs,string,supervisor,supervisor_bridge,sys,timer,unicode,unicode_util,uri_string,win32reg,zip]},{registered,[timer_server,rsh_starter,take_over_monitor,pool_master,dets]},{applications,[kernel]},{included_applications,[]},{env,[]},{maxT,infinity},{maxP,infinity}]}]}}
{progress,applications_loaded}
{apply,{application,start_boot,[kernel,permanent]}}
Erlang/OTP 22 [erts-10.7.2.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [dtrace]

{apply,{application,start_boot,[stdlib,permanent]}}
{apply,{c,erlangrc,[]}}
{progress,started}
Eshell V10.7.2.1  (abort with ^G)
1> <<214,82,31,217,30,215,232,232,193,110,25,222,249,130,35,78>>


Interestingly, the above fails when my vm.args is passed in (also tried passing in vm.args.example from the package)

Failure #1 (/usr/local/lib/erlang22/bin/erl -init_debug -s crypto -eval 'io:format("~w~n",[crypto:strong_rand_bytes(16)]).' -args_file /usr/local/etc/couchdb3/vm.args) - Attached as erl_crash.dump2

Failure #2 (/usr/local/lib/erlang22/bin/erl -init_debug -args_file /usr/local/etc/couchdb3/vm.args) - Attached as erl_crash.dump3

I tried with my local.ini (attached) as well as the short example you provided, same result to both (attached as erl_crash.dump)

I am attaching pkg info, uname, etc.

This is on a fresh install from Digital Ocean btw. Let me know if you need more info.
Comment 9 Dave Cottlehuber freebsd_committer 2020-07-20 12:37:18 UTC
I spun up a 4 GB Memory / 80 GB Disk / AMS3 - FreeBSD 12.1 x64 zfs in
DO (2 cpu like your setup) and followed my notes above and can't
replicate the issue, even using your local.ini

Slogan: init terminating in do_boot ({function_clause,[{init,b2a,[[_]],[]},{init,do_boot,3,[]}]})

b2a is binary_to_atom IIRC.

So I'm guessing your vm.args has a typo. can you re-try it, using

$ head /dev/random |sha512 | cut -c 1-32
5ae21db1f620a07927f254e052154ca7

to generate a node name and a cookie?

please post the resulting vm.args and whether that works.

This one looks similar, but maybe there's already one running?

Sun Jul 19 17:53:20 2020
Slogan: System process <0.0.0> terminated: {function_clause,[{init,prepare_run_args,[{eval,[<<"io:format(\"~w~n\",[crypto:strong_rand_bytes(16)]).">>,<<"Each">>,<<"node">>,<<"in">>,<<"the">>,<<"system">>,<<"must">>,<<"have">>,<<"a">>,<<"unique">>,<<"name.">>,<<"These">>,<<"are">>,<<"specified">>,<<"through">>]}],[]},{init,map,2,[]},{init,map,2,[]},{init,boot,1,[]}]}
Comment 10 Dave Cottlehuber freebsd_committer 2020-07-26 09:40:34 UTC
Erlang stack traces imply a local config file error or typo.
per previous comments, vm.args and any settings (specific DO
config for example) are needed to reproduce this.

I can't replicate this on a vanilla DO box with FreeBSD 12.1R
amd64p7 + couchdb3 from packages, Unless precise reproducible
steps are provided, or I get a bug report from elsewhere in
the community, I'll close this in a week or so.
Comment 11 Dave Cottlehuber freebsd_committer 2020-08-01 15:38:42 UTC
Mike, please re-open this if you can share enough info to reproduce it.