Summary: net-p2p/c-lightning 26.06.1 (and 25.12.1_2) build and install cleanly but fail to start: lightning_hsmd aborts with FATAL SIGNAL 6 the moment it is spawned with its message fd, and lightningd exits with HSM sent unknown message type. This reproduces on a brand-new node (freshly generated hsm_secret, empty database, regtest, no bitcoind) so it is not related to any user data, wallet state, or migration. The last working version is 25.09, which runs on the identical host and jail; downgrading to it immediately restores service. Environment: - Host: FreeBSD 15.1-RELEASE-p1, amd64 (kernel/userland/running all -p1), root on ZFS - Container: BastilleBSD thin jail, base r15.1-RELEASE - Hardware: Zimablade (Intel N-series CPU) - Good pkg: c-lightning-25.09 (pinned via pkg lock) - Failing pkgs: c-lightning-25.12.1_2, c-lightning-26.06.1 (default quarterly/latest repo) Reproduction (from scratch — no user data): bastille create clntest 15.1-RELEASE 10.17.89.9 bastille0 bastille cmd clntest pkg install -y c-lightning # 26.06.1 bastille cmd clntest mkdir -p /tmp/clntest bastille cmd clntest chown c-lightning:c-lightning /tmp/clntest bastille cmd clntest su -m c-lightning -c \ 'lightningd --network=regtest --lightning-dir=/tmp/clntest --log-level=debug' Observed: INFO lightningd: v26.06.1 ... (all bundled plugins start OK) ... DEBUG lightningd: testing /usr/local/.../lightning_hsmd <- hsmd SELF-TEST PASSES DEBUG hsmd: pid 8522, msgfd 70 <- real spawn w/ message fd lightning_hsmd: FATAL SIGNAL 6 (version v26.06.1) <- aborts immediately 0x... _thr_kill 0x... __raise 0x... abort ... (rest of backtrace is address-only; hsmd binary is stripped) ... lightningd: HSM sent unknown message type /usr/local/etc/rc.d/lightningd: WARNING: failed to start lightningd The abort precedes any Bitcoin-backend connection, so it is independent of network/RPC/Tor config. On our production node, downgrading (pkg install -f c-lightning-25.09) starts normally with the same node id and channels; the wallet DB is never migrated because the abort happens before the DB is opened. Analysis: HSM sent unknown message type means lightningd received a wire message from hsmd it could not parse. Notably the lightning_hsmd self-test succeeds but the real subdaemon aborts as soon as it is handed its message fd and processes the first hsm_init message. That points at the hsmd message/fd-handling path (a wire/ABI or fd-passing mismatch) introduced somewhere in the 25.09 → 25.12 window and still present in 26.06.1 — not data corruption, config, or environment (a fresh jail fails too). Binary / crash details: lightning_hsmd is stripped, so CLN's own crash handler prints the only backtrace available (address-only). The frame offsets are stable and identical across every run (production upgrade and the clean clntest jail), so they can be resolved with addr2line against the official package binary: lightning_hsmd offsets (in-binary): 0x374a24 0x35d927 0x395193 0x39508c 0x3968ca 0x35d026 file: ELF 64-bit LSB executable, x86-64, FreeBSD 15.0 (1500068), dynamically linked, stripped ldd: libm.so.5, libsqlite3.so.0, libsodium.so.26, libz.so.6, libgcc_s.so.1, libc.so.7, libthr.so.3, libsys.so.7 No core is produced (CLN's signal handler prints its backtrace and exits). Happy to provide a full debug log, or to build the port with debug symbols / run under a debugger if a symbolized backtrace would help. Requests: - Can the maintainer reproduce a fresh-node (regtest) start of 26.06.1 on 15.1? - A build of hsmd with debug symbols would let us pin the exact abort site — happy to run it. - Can provide pkg info -D c-lightning, ldd of lightning_hsmd, full debug log, and a symbolized backtrace on request.
ROOT CAUSE FOUND: hsmd calls mlock() on hsm_secret, which fails in a jail unless allow.mlock is set, and CLN aborts rather than degrading. I rebuilt the port with WITH_DEBUG=yes STRIP= and the DEBUG option to get a symbolized backtrace. The abort site is unambiguous: lightning_hsmd: FATAL SIGNAL 6 (version v26.06.6) 0x37af22 send_backtrace common/daemon.c:38 0x37b59e crashdump common/daemon.c:83 ... 0x825aa9918 abort+0x48 0x383372 mlock_tal_memory common/utils.c:104 0x364b53 load_hsm hsmd/hsmd.c:465 0x3641a7 init_hsm hsmd/hsmd.c:556 0x363c17 handle_client hsmd/hsmd.c:749 0x3a9b15 next_plan ccan/ccan/io/io.c:60 0x3aa7ec do_plan ccan/ccan/io/io.c:422 0x3aa61d io_ready ccan/ccan/io/io.c:439 0x3ac526 io_loop ccan/ccan/io/poll.c:470 0x363629 main hsmd/hsmd.c:893 hsmd loads hsm_secret and calls mlock() on it to keep the key out of swap. Inside a FreeBSD jail, mlock() is gated by the allow.mlock jail parameter, which is off by default. The lock fails and CLN aborts. This is a jail configuration requirement, not a build or packaging defect, and not the wire/ABI mismatch I originally suspected. Please disregard the fd-passing analysis in the original report. CONFIRMATION Host sysctl: security.bsd.unprivileged_mlock: 1 Jail parameters (BastilleBSD default, no allow.mlock set): # jls -j clntest -n | tr ' ' '\n' | grep mlock allow.nomlock Adding allow.mlock to the jail and restarting it lets the stock packaged 26.06.6 start normally -- no rebuild, no patch, no port change: # bastille config clntest set allow.mlock 1 # bastille restart clntest # jls -j clntest -n | tr ' ' '\n' | grep mlock allow.mlock Same unmodified binary package as before (Options: DEBUG off, RUST on), same clean regtest node: DEBUG hsmd: pid 29287, msgfd 70 DEBUG hsmd: capability +WIRE_HSMD_CHECK_PUBKEY ... INFO lightningd: Using BIP86 for new addresses, BIP32 for channels INFO lightningd: Creating database DEBUG connectd: pid 29288, msgfd 74 UNUSUAL hsmd: HSM: created new hsm_secret file DEBUG hsmd: new_client: 0 DEBUG connectd: Created listener on 0.0.0.0:19846 It then exits only with "The Bitcoin backend died", which is expected -- there is no bitcoind in this throwaway jail. WHY IT LOOKED LIKE A VERSION REGRESSION CLN added secret-memory locking in the 25.09 -> 25.12 window, which is exactly the boundary where this started for us. That also explains every other symptom: 25.09 works; 25.12.1_2, 26.06.1 and 26.06.6 all abort identically the lightning_hsmd --test self-check passes, because it never loads hsm_secret the abort lands the instant hsmd receives its real message fd, i.e. the first hsm_init -> load_hsm a brand-new regtest node fails too, because it still creates and locks a secret the in-binary crash offsets are byte-for-byte identical between 26.06.1 and 26.06.6, because nothing in this path changed between them "lightningd: HSM sent unknown message type" is a red herring; it is lightningd misreading an already-dead subdaemon VERSIONS TESTED ON THIS HOST 25.09 FreeBSD 14.4 / 15.1 works 25.12.1_2 FreeBSD 14.4 hsmd SIGABRT 26.06.1 FreeBSD 15.1 hsmd SIGABRT (quarterly) 26.06.6 FreeBSD 15.1 hsmd SIGABRT (latest) All in BastilleBSD thin jails without allow.mlock. SUGGESTED PORT CHANGE Since c-lightning is very commonly run in a jail, and the failure mode is a bare SIGABRT with no diagnostic, a pkg-message noting that jails require allow.mlock (from 25.12 onward) would save the next person a long investigation. Happy to draft one if that is useful. Separately, aborting on mlock() failure with no error message seems worth raising upstream -- containerized Linux deployments hitting RLIMIT_MEMLOCK would fail the same opaque way. I can file that with ElementsProject/lightning if you agree it is out of scope for the port.
Matt, thank you for the excellent analysis! > Since c-lightning is very commonly run in a jail, and the failure mode is a bare SIGABRT with no diagnostic, a pkg-message noting that jails require allow.mlock (from 25.12 onward) would save the next person a long investigation. Happy to draft one if that is useful. Yes, I would add that to pkg-message. I am unsure why it works in my environment - cln in a jail and mlock is not explicitly enabled. > Separately, aborting on mlock() failure with no error message seems worth raising upstream -- containerized Linux deployments hitting RLIMIT_MEMLOCK would fail the same opaque way. I can file that with ElementsProject/lightning if you agree it is out of scope for the port. Yes, that is worthy of an upstream patch (or pull request on github). I can include the patch in the FreeBSD port before it gets included in an official upstream release. Cheerz!