After updating ports/packages to py38-salt 3003.2, salt minion do not start anymore _on a subset of systems_. All hosts affected running FreeBSD 13.0-RELENG-p4, most of the working/none-working salt_minion installations are running inside jails of the same OS flavor and version as the ports are taken from the very same source (simply to make clear that working/none-working jails/hosts share the same base OS and the same repo where the packages has been taken and installed from). Installed port is: py38-salt-3003.2 sysutils/py-salt Host's OS is FreeBSD 13.0-RELEASE-p4 releng/13.0-n244760-940681634ee amd64 The error is: [...] # service salt_minion restart salt_minion not running? (check /var/run/salt-minion.pid). Starting salt_minion. Traceback (most recent call last): File "/usr/local/bin/salt-minion", line 33, in <module> sys.exit(load_entry_point('salt==3003.2', 'console_scripts', 'salt-minion')()) File "/usr/local/bin/salt-minion", line 25, in importlib_load_entry_point return next(matches).load() StopIteration /usr/local/etc/rc.d/salt_minion: WARNING: failed to start salt_minion According to some findings on the net, this could be related to some problems with libraries, but I'm unable to discover the differences without disturbing or destroying services we rely on those specific hosts. Jails/Hosts with lots of py38- related installations, like several Apache24 services, seem to fail, while, for instance, Apache24-free servers (only Icinga2 as master installed) work fine. The source on the net referred to above is: https://github.com/saltstack/salt/issues/60206 It is Linux, not FreeBSD, but it might be a hint for the maintainer.
The problem is still present and doesn't vanish with a complete reinstall of the port.
I'm now seeing this in 3004.1 and its also affecting the master. [root@salt /usr/local/etc/salt]# /usr/local/etc/rc.d/salt_minion start Starting salt_minion. Traceback (most recent call last): File "/usr/local/bin/salt-minion", line 33, in <module> sys.exit(load_entry_point('salt==3004.1', 'console_scripts', 'salt-minion')()) File "/usr/local/bin/salt-minion", line 25, in importlib_load_entry_point return next(matches).load() StopIteration /usr/local/etc/rc.d/salt_minion: WARNING: failed to start salt_minion [root@salt /usr/local/etc/salt]# /usr/local/etc/rc.d/salt_master start Starting salt_master. Traceback (most recent call last): File "/usr/local/bin/salt-master", line 33, in <module> sys.exit(load_entry_point('salt==3004.1', 'console_scripts', 'salt-master')()) File "/usr/local/bin/salt-master", line 25, in importlib_load_entry_point return next(matches).load() StopIteration /usr/local/etc/rc.d/salt_master: WARNING: failed to start salt_master
(In reply to Michele Possamai from comment #2) Replying to my own comment as I (kinda) fixed it. It happened after this update: py38-salt upgraded: 3004_1 -> 3004.1_1 I rolled back by installing the old version from /var/cache/pkg and immediately it started working again.
Read carefully https://saltproject.io/security_announcements/salt-security-advisory-release/ NOTE: When upgrading your Salt infrastructure, first upgrade your Salt master packages before upgrading your Salt minion packages. Upgrading the minion packages first could result in loss of functionality.
(In reply to Kirill Ponomarev from comment #4) While I agree that not being able to start it at all is a very effective way to mitigate a security hole, I'm not sure it's the best way... I've upgraded the minions on other machines and they all run fine so something is probably off with my saltmaster-server. It's a shame the error message is so incredibly useless.
What is the solution? Did I miss something here? This problem occurs even with the latest ports and with fresh installations. SALT master is running on 13.0-p11jail, as well as the mutually installed minions. The problem seems to be random among jails.
In my case my python setup was apparently messed up. I removed everything python related. Removed leftover python stuff manually and reinstalled everything. This fixed it for me. Maybe it was something left from experimenting with pip outside a venv, I don't know. But it's obvious it was python related (for me). So not really a good solution and I understand not everyone can just delete all python stuff like that. It worked for me though and I didn't look any further.
Id did a similar procedure - deleteing python and all stuff related to it (pakcages, libs), but it did not change anything. That was the first attempt to solve the problem. Then, I installed another jail on the host and did a completely fresh install of saltstack there - and got the same weird result. Another jail did act as expected. The situation happend on 13.0-RELENG, for the record.
As mentioned by some comments, one has to delete manually leftovers from earlier installations and/or misfits in upgrade procedures. I did rely on pkg - and pkg left some remnants in /usr/local/lib/site-packages/ which I deleted then manually. Also, there were leftover folders from python3.7 in /usr/local/lib, also not taken care of by pkg - I had to delete them, too. After a time consuming manual delete-orgy and reinstallation of py-salt, everything worked again as expected! Thanks for assitance, case closed :-)