Bug 248234 - devel/py-buildbot-worker home directory environment incorrect
Summary: devel/py-buildbot-worker home directory environment incorrect
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Muhammad Moinur Rahman
URL:
Keywords:
Depends on: 248233
Blocks:
  Show dependency treegraph
 
Reported: 2020-07-24 03:05 UTC by ari
Modified: 2022-12-14 14:21 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description ari 2020-07-24 03:05:47 UTC
After installing the port in a new FreeBSD 12.1 jail, and starting it with the default 'buildbot' user, the buildbot environment contains a HOME pointing to /root instead of /home/buildbot.

Passing --uid to twistd isn't setting the home directory to the correct place.

A more FreeBSD approach to running with the correct user is to set the variable:

buildbot_worker_user

buildbot_worker_uid isn't the best way to do it, since that doesn't cause rc.subr to properly su to the correct user and twistd is itself launched as root.

My workaround now is to set:

buildbot_worker_user="buildbot"
buildbot_worker_env="HOME=/home/buildbot USER=buildbot"
Comment 1 Rene Ladan freebsd_committer freebsd_triage 2022-03-07 19:54:56 UTC
Maintainer reset.
Comment 2 Muhammad Moinur Rahman freebsd_committer freebsd_triage 2022-03-14 06:14:07 UTC
(In reply to ari from comment #0)
Can you please confirm whether if instead of this line:
buildbot_worker_env="HOME=/home/buildbot USER=buildbot"

Setting it like this:
buildbot_worker_basedir="/home/buildbot"

Yields similar result?
Comment 3 ari 2022-03-14 06:30:27 UTC
I've always had buildbot_worker_basedir set to another directory for where the builds go. But I don't think it changes the home directory.

/usr/local/etc/buildbot-worker is a strange default for buildbot_worker_basedir since that contains all the build files, not just the configuration. So I set it to /var/builds which makes more sense for that kind of data.

I believe that basedir and the HOME env are unrelated to each other.
Comment 4 Muhammad Moinur Rahman freebsd_committer freebsd_triage 2022-03-14 06:37:37 UTC
(In reply to ari from comment #3)
What I am unsure about this is that there is no var called "buildbot_worker_env" in the rc script. So how come setting/unsetting this variable actually affects the startup? Are you sure that you don't have any extra patched applied to the script? Cause:

grep buildbot_worker_env /usr/local/etc/rc.d/buildbot-worker

returns null.
Comment 5 ari 2022-03-14 07:39:04 UTC
# grep ${name}_env /etc/rc.subr


The bad thing that happens is that buildbot worker tries to
Comment 6 ari 2022-03-14 07:41:12 UTC
# grep ${name}_env /etc/rc.subr

The bad thing that happens for buildbot worker (I changed the title of this ticket to reflect that the worker is the real problem here) is that many build scripts try to save cache files into the user home directory.

For example, gradle puts its cache to ~/.gradle
Comment 7 Muhammad Moinur Rahman freebsd_committer freebsd_triage 2022-03-14 07:51:41 UTC
(In reply to ari from comment #6)
Ah okiz. So I am going to create a user which isn't the problem. The problem is it's advisable that when we are creating users those are used by ports should not go to /usr/home or /home. Preferred location is under /var/db. So can you make a quick test and let me know that if you change the $HOME to /var/db/buildbot in both passdb and the rc.conf if it works properly. Then I can create a real quick patch to create the users and incorporate them. I will also fix the bug#262545 in one go with that. This will also have effect on bug#248233.
Comment 8 Muhammad Moinur Rahman freebsd_committer freebsd_triage 2022-03-14 08:13:44 UTC
(In reply to ari from comment #0)
One more thing is as you have mentioned about twistd is not doing proper su to the user have you ever tried it with --euid with twistd. It's not yet in the rc script but I need to test it. 

Sorry that I am discussing more but I am just planning to switch to buildbot from Gitlab runners and do not have clear understanding of the internals of twistd too much.
Comment 9 ari 2022-03-14 22:27:42 UTC
I've never tried -euid and I don't have that much experience with twisted myself. Just enough to get it launched and hope it works. In my own projects I've used Flask.

Don't apologise. I appreciate the work you are doing and its good to have a plan first.
Comment 10 commit-hook freebsd_committer freebsd_triage 2022-03-17 06:34:30 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=dc041931d95ad097170e3055f6f11ac2523bf32c

commit dc041931d95ad097170e3055f6f11ac2523bf32c
Author:     Muhammad Moinur Rahman <bofh@FreeBSD.org>
AuthorDate: 2022-03-17 06:23:40 +0000
Commit:     Muhammad Moinur Rahman <bofh@FreeBSD.org>
CommitDate: 2022-03-17 06:33:32 +0000

    devel/py-buildbot-worker: Add USERS/GROUPS

    - Fix missing dependency of www/py-autobahn [1]

    PR:             248234 262545 [1]
    Reported by:    ari@ish.com.au

 devel/py-buildbot-worker/Makefile | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
Comment 11 Muhammad Moinur Rahman freebsd_committer freebsd_triage 2022-03-17 06:39:19 UTC
Although I have committed some changes but it actually doesn't resolve the problem mentioned here fully. For the actual fix we have to rewrite the entire rc script. For an idea you can have a look at the rc script of buildbot itself. Currently there is a helper script named buildbot-worker which actually should be used to start/stop the service rather than using it with twisted. Currently I am a short of time and will have a look at this at the next weekend(not the coming one). Keeping this ticket open for now.
Comment 12 commit-hook freebsd_committer freebsd_triage 2022-12-14 14:19:48 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=b6471b43913ddd7d12d07bc3bfa2c2ab3a4aac7b

commit b6471b43913ddd7d12d07bc3bfa2c2ab3a4aac7b
Author:     Muhammad Moinur Rahman <bofh@FreeBSD.org>
AuthorDate: 2022-12-14 14:08:22 +0000
Commit:     Muhammad Moinur Rahman <bofh@FreeBSD.org>
CommitDate: 2022-12-14 14:18:51 +0000

    devel/py-buildbot: Update version 3.5.0=>3.7.0

    - Use the existing buildbot users and groups
    - Refactor the rc script to run as buildbot user and pointing the
      basedir to the home directory of buildbot user [1]

    PR:             248234 [1]
    Reported by:    ari@ish.com.au [1]

 devel/py-buildbot/Makefile          |  6 +++++-
 devel/py-buildbot/distinfo          |  6 +++---
 devel/py-buildbot/files/buildbot.in | 16 ++++++++++++----
 3 files changed, 20 insertions(+), 8 deletions(-)
Comment 13 commit-hook freebsd_committer freebsd_triage 2022-12-14 14:19:50 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=0620448973d04bd533a728538da3f8184c1e4ab1

commit 0620448973d04bd533a728538da3f8184c1e4ab1
Author:     Muhammad Moinur Rahman <bofh@FreeBSD.org>
AuthorDate: 2022-12-14 14:11:20 +0000
Commit:     Muhammad Moinur Rahman <bofh@FreeBSD.org>
CommitDate: 2022-12-14 14:18:51 +0000

    devel/py-buildbot-worker: Update version 3.5.0=>3.7.0

    - Use the existing buildbot users and groups
    - Refactor the rc script to use the existing users. Please be informed
      that buildbot_worker_{uid|gid} is no longer the recommended rcvar. If
      a different user needs to be used the rcvar should be updated with
      buildbot_worker_user. Currently this defaults to buildbot and
      buildbot_worker_basedir defaults to a subfolder named workers under
      the home directory of buildbot that is /var/db/buildbot [1]
    - Refactor the rc script to use the program as the command rather than
      using twistd as per upstream documentation

    PR:             248234 [1]
    Reported by:    ari@ish.com.au [1]

 devel/py-buildbot-worker/Makefile                 |  3 +-
 devel/py-buildbot-worker/distinfo                 |  6 +-
 devel/py-buildbot-worker/files/buildbot-worker.in | 79 +++++++++++++----------
 3 files changed, 49 insertions(+), 39 deletions(-)
Comment 14 Muhammad Moinur Rahman freebsd_committer freebsd_triage 2022-12-14 14:21:41 UTC
These commits should fix the problem you have mentioned. Please check the commit logs and let me know or reopen if the problem still persists.