Bug 253857 - devel/jenkins-lts: Aborted builds leave processes behind
Summary: devel/jenkins-lts: Aborted builds leave processes behind
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Li-Wen Hsu
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-02-25 20:58 UTC by Michael Gmelin
Modified: 2022-04-05 14:13 UTC (History)
2 users (show)

See Also:
bugzilla: maintainer-feedback? (lwhsu)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Gmelin freebsd_committer freebsd_triage 2021-02-25 20:58:57 UTC
Jenkins makes use of a feature called the ProcessTreeKiller[0] to kill processes when aborting a build (e.g., timeout, manual abort, etc.). The ProcessTreeKiller checks the environment of all processes to see if they contain the BUILD_ID of the current build, and if they do, it kills them.

The ProcessTreeKiller doesn't support FreeBSD though, as it depends on being able to read the environment from procfs[1].

In theory, it should be able to modify the ProcessTree implementation in Jenkins to support this, so that it works as expected. Doing so might be quite some work (especially when trying to upstream a patch).

So if fixing the problem isn't possible, it would be good if the port would warn about it (and recommending a workaround).

I'm currently using the script below, called from within a shell build step, to emulate expected ProcessTreeKiller behavior:

build_cleanup.sh:
 
#!/bin/sh
#
# 2021 Michael Gmelin <grembo@FreeBSD.org>
#
# When aborting a build, Jenkins kills all processes containing
# the correct BUILD_ID in their environment using the
# ProcessTreeKiller.
#
# As the ProcessTreeKiller is not supported on FreeBSD,
# aborting a build can leave processes behind.
#
# This script mitigates this by waiting for the
# script it was executed by to end and then
# emulating ProcessTreeKiller behavior (check
# all processes if they're containing BUILD_ID=<build_id>
# in their environment.
#
# Example invocation from build script:
# (env -u BUILD_ID build_cleanup.sh "$$" "$BUILD_ID")&

pwait $1

for p in $(ps -xopid | grep -v PID); do
  if [ "$p" -ne "$$" ]; then
    procstat -e $p 2>/dev/null | tr " " "\n" | grep -q "BUILD_ID=$2"
    if [ $? -eq 0 ]; then
      kill $p
    fi
  fi
done

Example build.sh:

#!/bin/sh

(env -u BUILD_ID build_cleanup.sh "$$" "$BUILD_ID")&
exec ./build_all.sh

[0]https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller
[1]https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/util/ProcessTree.java
Comment 1 Ronald Klop 2021-09-05 10:05:39 UTC
Is this fixed here? 

https://github.com/jenkinsci/jenkins/pull/5563
Comment 2 Michael Gmelin freebsd_committer freebsd_triage 2022-04-05 14:13:14 UTC
(In reply to Ronald Klop from comment #1)

This looks very promising, indeed. I'll test it after updating jenkins-lts in the not-to-distant future locally and report back/potentially close this PR.