Bug 232146

Summary: mail/spamassassin: service status broken after reload
Product: Ports & Packages Reporter: Romain Tartière <romain>
Component: Individual Port(s)Assignee: Niclas Zeising <zeising>
Status: Closed FIXED    
Severity: Affects Many People CC: freebsd, mark, romain, solence
Priority: --- Keywords: needs-qa
Version: LatestFlags: bugzilla: maintainer-feedback? (zeising)
koobs: merge-quarterly?
Hardware: Any   
OS: Any   
Bug Depends on: 232501    
Bug Blocks:    

Description Romain Tartière freebsd_committer freebsd_triage 2018-10-10 07:31:56 UTC
Hi

The sa-spamd rc script allows to tell if the service is running or not with `service sa-spamd status`.

Unfortunately, after reloading the service, the command always says that the service is not running anymore:

λ service sa-spamd start
Starting spamd.
λ service sa-spamd status
spamd is running as pid 62822.
λ ps l 62822
UID   PID PPID CPU PRI NI    VSZ    RSS MWCHAN STAT TT     TIME COMMAND
  0 62822    1   0  20  0 203316 191940 select Ss    -  0:26,12 /usr/local/bin/perl -T -w /usr/local/bin/spamd -c --allow-tell --username=postfix -H /var/spool/postfix -d -r /var/run/spamd/spamd.pid
λ service sa-spamd reload
λ service sa-spamd status
spamd is not running.
λ ps l 62822
UID   PID PPID CPU PRI NI   VSZ   RSS MWCHAN STAT TT     TIME COMMAND
  0 62822    1   0  21  0 40200 34704 nanslp Ss    -  0:26,50 /usr/local/bin/perl5 -T -w /usr/local/bin/spamd -c --allow-tell --username=postfix -H /var/spool/postfix -d -r /var/run/spamd/spamd.pid


It looks like the command is changing (perl -> perl5), which seems to break the system in charge of finding the right process.

This is at least on 11.2-STABLE.
Comment 1 Romain Tartière freebsd_committer freebsd_triage 2018-10-10 07:43:30 UTC
Changing the shebang and $command_interpreter to "/usr/local/bin/perl5" in the rc-script does not get around the issue.

My spamd configuration is:
spamd_enable="YES"
spamd_flags="-c --allow-tell --username=postfix -H /var/spool/postfix"
Comment 2 Robin 2018-10-12 19:02:30 UTC
I have the same issue, it doesn't affect just you.
Comment 3 Helge Oldach 2018-10-21 05:19:37 UTC
Same issue here.

The culprit seems to be that _proccheck (in _find_processes() from /etc/rc.subr) now compares $command_interpreter (specified in the sa-spamd rc script) but that doesn't match with the command line of the process:

# ps ax | fgrep spamd
69955  -  Ss       1:15.91 /usr/local/bin/perl5.26.2 -T -w /usr/local/bin/spamd
70304  -  I        0:35.68 spamd child (perl5.26.2)
70305  -  I        0:13.40 spamd child (perl5.26.2)
78673  1  S+       0:00.01 fgrep spamd
# fgrep command_inter /usr/local/etc/rc.d/sa-spamd
command_interpreter="/usr/local/bin/perl"
#

Although /usr/local/bin/perl5.26.2 and /usr/local/bin/perl are in fact hardlinked this is a textual mismatch.

I seem to recall that before the recent mail/spamassassin update the command interpreter as seen on the command line was /usr/local/bin/perl and not $^X, so it seems below logic in spamd (sub do_sighup_restart) breaks it:

  # ensure we re-run spamd using the right perl interpreter, and
  # with the right switches (taint mode and warnings) (bug 5255)
  my $perl = untaint_var($^X);
  my @execs = ( $perl, "-T", "-w", $ORIG_ARG0, @ORIG_ARGV );
Comment 4 Helge Oldach 2018-10-21 05:26:09 UTC
Changing the shebang line of /usr/local/bin/spamd to:

#!/usr/local/bin/perl5.26.2 -T -w

fixes it.

Should we push the $^X logic into the port's Makefile?
Comment 5 Helge Oldach 2018-10-21 05:42:57 UTC
and add the same to command_interpreter in rc.d/sa-spamd of course.
Comment 6 Helge Oldach 2018-10-21 10:04:20 UTC
Hmmm. The problem ist more complex actually. All potential invocations of perl, perl5, or perl5.26.2 appear to show up over time in the command line that is derived from $^X.

I guess it has to do with the fact that perl, perl5 and perl5.26.2 are hard linked. The behavior of $^X actually depends on which of the three hard links had been used /last/ to run perl. Once all three had been used, the last one remains sticky:

# reboot straight into single-user, don't run anything 
# perl -e 'print $^X, "\n"'
/usr/local/bin/perl
# perl5 -e 'print $^X, "\n"'
/usr/local/bin/perl5
# perl5.26.2 -e 'print $^X, "\n"'
/usr/local/bin/perl5.26.2
# perl -e 'print $^X, "\n"'
/usr/local/bin/perl5.26.2
# perl5 -e 'print $^X, "\n"'
/usr/local/bin/perl5.26.2
# perl5.26.2 -e 'print $^X, "\n"'
/usr/local/bin/perl5.26.2
# perl -e 'print $^X, "\n"'
/usr/local/bin/perl5.26.2
# perl5 -e 'print $^X, "\n"'
/usr/local/bin/perl5.26.2
# perl5.26.2 -e 'print $^X, "\n"'
/usr/local/bin/perl5.26.2

# reboot straight into single-user, don't run anything 
# perl5.26.2 -e 'print $^X, "\n"'
/usr/local/bin/perl5.26.2
# perl5 -e 'print $^X, "\n"'
/usr/local/bin/perl5
# perl -e 'print $^X, "\n"'
/usr/local/bin/perl
# perl5.26.2 -e 'print $^X, "\n"'
/usr/local/bin/perl
# perl5 -e 'print $^X, "\n"'
/usr/local/bin/perl
# perl -e 'print $^X, "\n"'
/usr/local/bin/perl
# perl5.26.2 -e 'print $^X, "\n"'
/usr/local/bin/perl
# perl5 -e 'print $^X, "\n"'
/usr/local/bin/perl
# perl -e 'print $^X, "\n"'
/usr/local/bin/perl

# reboot straight into single-user, don't run anything 
# perl -e 'print $^X, "\n"'
/usr/local/bin/perl
# perl5.26.2 -e 'print $^X, "\n"'
/usr/local/bin/perl5.26.2
# perl5 -e 'print $^X, "\n"'
/usr/local/bin/perl5
# perl -e 'print $^X, "\n"'
/usr/local/bin/perl5
# perl5.26.2 -e 'print $^X, "\n"'
/usr/local/bin/perl5
# perl5 -e 'print $^X, "\n"'
/usr/local/bin/perl5
# perl -e 'print $^X, "\n"'
/usr/local/bin/perl5
# perl5.26.2 -e 'print $^X, "\n"'
/usr/local/bin/perl5
# perl5 -e 'print $^X, "\n"'
/usr/local/bin/perl5

In other words, use of $^X is non-deterministic in FreeBSD.

I suspect the canonical fix would be to replace the perl hard links with soft links perl -> perl5 -> perl5.26.2. That needs to be done int the perl ports. Alternatively, teach rc.subr how to deal with shebang program names pointing to hard-linked commands?
Comment 7 Niclas Zeising freebsd_committer freebsd_triage 2018-10-22 08:06:54 UTC
Hi!
Thank you very much for helping to debug this!
I have to look at the details a bit, but I'm not sure it's possible to work around this in spamassassin, unfortunately, apart from maybe changing the shebang line.
What happens if you run it as /usr/bin/env perl?
Comment 8 Mark Hills 2018-10-29 16:49:05 UTC
I concur with everything in this report; I was about to file the same bug which has been on my system for what seems like a few weeks. I traced it to the same piece of rc.subr.

# ps ax | grep spamd
22340  0  R+       0:00.00 grep spamd

# service sa-spamd start
Starting spamd.

# service sa-spamd status
spamd is running as pid 22348.

# ps ax | grep spamd
22348  -  Ss       0:01.57 /usr/local/bin/perl -T -w /usr/local/bin/spamd -u spamd -H /var/spool/spamd -d -r /var/run/spamd/spamd.pid
22349  -  S        0:00.00 spamd child (perl)
22350  -  S        0:00.00 spamd child (perl)
22359  0  S+       0:00.00 grep spamd

# service sa-spamd reload

# service sa-spamd status
spamd is not running.

# ps ax | grep spamd
22369  -  Ss       0:01.55 /usr/local/bin/perl5.26.2 -T -w /usr/local/bin/spamd -u spamd -H /var/spool/spamd -d -r /var/run/spamd/spamd.pid
22370  -  S        0:00.00 spamd child (perl5.26.2)
22371  -  S        0:00.00 spamd child (perl5.26.2)
22379  0  R+       0:00.00 grep spamd

# cat /var/run/spamd/spamd.pid 
22369
Comment 9 Niclas Zeising freebsd_committer freebsd_triage 2018-11-01 11:16:16 UTC
Perl was just updated to install symlinks instead of hard links.  Hopefully that fixes the issue.
Comment 10 Robin 2018-11-03 09:44:35 UTC
I just updated Perl to perl5-5.28.0_1, but that didn't fix it for me. The issue is the same.
Comment 11 Niclas Zeising freebsd_committer freebsd_triage 2019-01-03 02:20:25 UTC
Is this still an issue?

Perl has been updated with a potential fix for this.
Comment 12 Romain Tartière freebsd_committer freebsd_triage 2019-01-03 03:29:15 UTC
I just re-checked with perl5-5.28.1 and the problem is still present.

In case you are not speaking about the update of lang/perl5.28 to 5.28.1, please point me to the right commit so that I can check if I have it (ports where built some days ago, a recent change will not be in).

Thanks!
Comment 13 Helge Oldach 2019-01-03 06:52:22 UTC
(In reply to Romain Tartière from comment #12)
I cannot agree. Please validate that your perl executables are softlinked like this:

# ls -la /usr/local/bin/perl*
-rwxr-xr-x  1 root  wheel  15424 Dec 30 15:35 /usr/local/bin/perl*
lrwxr-xr-x  1 root  wheel      4 Dec 30 15:35 /usr/local/bin/perl5@ -> perl
lrwxr-xr-x  1 root  wheel      4 Dec 30 15:35 /usr/local/bin/perl5.28.1@ -> perl

If you still see had links as before Nov 18 I suspect there is something wrong in your setup...
Comment 14 Helge Oldach 2019-01-03 07:09:53 UTC
(In reply to Romain Tartière from comment #12)
ports r483655
Comment 15 Romain Tartière freebsd_committer freebsd_triage 2019-01-03 17:00:43 UTC
I do have symlinks as expected:

-rwxr-xr-x  1 root  wheel   7800 18 déc.  13:28 /usr/local/bin/perl*
lrwxr-xr-x  1 root  wheel      4 18 déc.  13:28 /usr/local/bin/perl5@ -> perl
lrwxr-xr-x  1 root  wheel      4 18 déc.  13:28 /usr/local/bin/perl5.28.1@ -> perl




However, I just see that on service reload, the pid file is erased:

λ cat /var/run/spamd/spamd.pid 
86731
λ service sa-spamd status
spamd is running as pid 86731.
λ service sa-spamd reload
λ cat /var/run/spamd/spamd.pid
cat: /var/run/spamd/spamd.pid: No such file or directory
Comment 16 Helge Oldach 2019-01-03 19:35:28 UTC
(In reply to Romain Tartière from comment #15)
Perl looks good indeed. Perhaps you just might need to wait for a couple of seconds, as spamd is pretty heavy?

# cat /var/run/spamd/spamd.pid
30256
# service sa-spamd reload
# cat /var/run/spamd/spamd.pid
cat: /var/run/spamd/spamd.pid: No such file or directory
# sleep 30
# cat /var/run/spamd/spamd.pid
cat: /var/run/spamd/spamd.pid: No such file or directory
# cat /var/run/spamd/spamd.pid
30493
#
Comment 17 Romain Tartière freebsd_committer freebsd_triage 2019-01-04 02:07:10 UTC
Wow!  Yes, the "solution" is to wait :-)  Thanks!

Closing the issue since the root issue is fixed.  Thank you all for your efforts for fixing the problem!