Bug 258053 - sysutils/py-salt: etc/rc.d/salt_minion: WARNING: failed to start salt_minion
Summary: sysutils/py-salt: etc/rc.d/salt_minion: WARNING: failed to start salt_minion
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Kirill Ponomarev
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-08-26 06:52 UTC by O. Hartmann
Modified: 2022-05-06 09:56 UTC (History)
2 users (show)

See Also:
bugzilla: maintainer-feedback? (christer.edwards)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description O. Hartmann 2021-08-26 06:52:30 UTC
After updating ports/packages to py38-salt 3003.2, salt minion do not start anymore _on a subset of systems_. All hosts affected running FreeBSD 13.0-RELENG-p4, most of the working/none-working salt_minion installations are running inside jails of the same OS flavor and version as the ports are taken from the very same source (simply to make clear that working/none-working jails/hosts share the same base OS and the same repo where the packages has been taken and installed from).

Installed port is:
py38-salt-3003.2 sysutils/py-salt

Host's OS is
FreeBSD 13.0-RELEASE-p4 releng/13.0-n244760-940681634ee amd64

The error is:
[...]
# service salt_minion restart
salt_minion not running? (check /var/run/salt-minion.pid).
Starting salt_minion.
Traceback (most recent call last):
  File "/usr/local/bin/salt-minion", line 33, in <module>
    sys.exit(load_entry_point('salt==3003.2', 'console_scripts', 'salt-minion')())
  File "/usr/local/bin/salt-minion", line 25, in importlib_load_entry_point
    return next(matches).load()
StopIteration
/usr/local/etc/rc.d/salt_minion: WARNING: failed to start salt_minion

According to some findings on the net, this could be related to some problems with libraries, but I'm unable to discover the differences without disturbing or destroying services we rely on those specific hosts. Jails/Hosts with lots of py38- related installations, like several Apache24 services, seem to fail, while, for instance, Apache24-free servers (only Icinga2 as master installed) work fine. The source on the net referred to above is:

https://github.com/saltstack/salt/issues/60206

It is Linux, not FreeBSD, but it might be a hint for the maintainer.
Comment 1 O. Hartmann 2021-09-22 05:03:21 UTC
The problem is still present and doesn't vanish with a complete reinstall of the port.
Comment 2 Michele Possamai 2022-04-02 20:45:06 UTC
I'm now seeing this in 3004.1 and its also affecting the master.


[root@salt /usr/local/etc/salt]# /usr/local/etc/rc.d/salt_minion start
Starting salt_minion.
Traceback (most recent call last):
  File "/usr/local/bin/salt-minion", line 33, in <module>
    sys.exit(load_entry_point('salt==3004.1', 'console_scripts', 'salt-minion')())
  File "/usr/local/bin/salt-minion", line 25, in importlib_load_entry_point
    return next(matches).load()
StopIteration
/usr/local/etc/rc.d/salt_minion: WARNING: failed to start salt_minion

[root@salt /usr/local/etc/salt]# /usr/local/etc/rc.d/salt_master start
Starting salt_master.
Traceback (most recent call last):
  File "/usr/local/bin/salt-master", line 33, in <module>
    sys.exit(load_entry_point('salt==3004.1', 'console_scripts', 'salt-master')())
  File "/usr/local/bin/salt-master", line 25, in importlib_load_entry_point
    return next(matches).load()
StopIteration
/usr/local/etc/rc.d/salt_master: WARNING: failed to start salt_master
Comment 3 Michele Possamai 2022-04-02 20:57:16 UTC
(In reply to Michele Possamai from comment #2)
Replying to my own comment as I (kinda) fixed it.

It happened after this update:
py38-salt upgraded: 3004_1 -> 3004.1_1

I rolled back by installing the old version from /var/cache/pkg and immediately it started working again.
Comment 4 Kirill Ponomarev freebsd_committer freebsd_triage 2022-04-02 21:04:26 UTC
Read carefully https://saltproject.io/security_announcements/salt-security-advisory-release/

NOTE: When upgrading your Salt infrastructure, first upgrade your Salt master packages before upgrading your Salt minion packages. Upgrading the minion packages first could result in loss of functionality.
Comment 5 Michele Possamai 2022-04-03 07:41:29 UTC
(In reply to Kirill Ponomarev from comment #4)

While I agree that not being able to start it at all is a very effective way to mitigate a security hole, I'm not sure it's the best way...

I've upgraded the minions on other machines and they all run fine so something is probably off with my saltmaster-server. It's a shame the error message is so incredibly useless.
Comment 6 O. Hartmann 2022-04-22 13:16:28 UTC
What is the solution? Did I miss something here? 

This problem occurs even with the latest ports and with fresh installations. SALT master is running on 13.0-p11jail, as well as the mutually installed minions.

The problem seems to be random among jails.
Comment 7 Michele Possamai 2022-04-22 15:28:08 UTC
In my case my python setup was apparently messed up.  
I removed everything python related. Removed leftover python stuff manually and reinstalled everything. This fixed it for me. Maybe it was something left from experimenting with pip outside a venv, I don't know. But it's obvious it was python related (for me).  

So not really a good solution and I understand not everyone can just delete all python stuff like that. It worked for me though and I didn't look any further.
Comment 8 O. Hartmann 2022-04-25 04:47:41 UTC
Id did a similar procedure - deleteing python and all stuff related to it (pakcages, libs), but it did not change anything. That was the first attempt to solve the problem. Then, I installed another jail on the host and did a completely fresh install of saltstack there - and got the same weird result. Another jail did act as expected.

The situation happend on 13.0-RELENG, for the record.
Comment 9 O. Hartmann 2022-05-06 09:56:01 UTC
As mentioned by some comments, one has to delete manually leftovers from earlier installations and/or misfits in upgrade procedures. I did rely on pkg - and pkg left some remnants in /usr/local/lib/site-packages/ which I deleted then manually. Also, there were leftover folders from python3.7 in /usr/local/lib, also not taken care of by pkg - I had to delete them, too.

After a time consuming manual delete-orgy and reinstallation of py-salt, everything worked again as expected!

Thanks for assitance,

case closed :-)