Bug 236840

Summary: textproc/elasticsearch6: Elasticsearch 6 service doesn't start at boot time with clear_tmp_enable="YES" in /etc/rc.conf
Product: Ports & Packages Reporter: Nemesis11 <felix>
Component: Individual Port(s)Assignee: Juraj Lutter <otis>
Status: Closed FIXED    
Severity: Affects Some People CC: elastic, felix, lwhsu, otis
Priority: --- Flags: bugzilla: maintainer-feedback? (elastic)
Version: Latest   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
Images showing the bug none

Description Nemesis11 2019-03-27 22:44:50 UTC
Created attachment 203199 [details]
Images showing the bug

I created a machine with FreeBSD 12.0 and the Elasticsearch 6 and, when we have the option "Clean the /tmp filesystem on system startup" (clear_tmp_enable="YES" in /etc/rc.conf) on, the Elasticsearch 6 service doesn’t start at boot time.
"Clean the /tmp filesystem on system startup" is a security option at install time, that can be enabled/disabled.
The problem happens because the Elasticsearch service starts soon after the Clean /tmp service. The java process tries to write on /tmp at the same time the clean /tmp service is deleting everything in /tmp.

Most of the time, there is no error on the console, at boot, and nothing is written in the logs. But one of the time, i saw the java process giving the error that it couldn't open file /tmp/elasticsearch.xxxxxxxx/gc.log in the console, at boot.

-------------------------------------------------------
Clearing /tmp.
OpenJDK 64-Bit Server VM Warning: Cannot open file /tmp/elasticsearch.wzc6rYAS/gc.log
Updating motd:
-------------------------------------------------------

Even when there is no error in the boot console, the Elasticsearch 6 service doesn't start at boot time with clear_tmp_enable="YES" in /etc/rc.conf.
After the boot sequence, i looked the status of the service, and the Elasticsearch 6 service was not running.

-------------------------------------------------------
felix@elasticforum2:~ % doas service elasticsearch status
elasticsearch is not running.
-------------------------------------------------------

After boot, i started the Elasticsearch 6 service manually, with "service elasticsearch start", it starts with no problem.
If, in /etc/rc.conf, if i comment the Clean /tmp service ( with #clear_tmp_enable="YES" ), the Elasticsearch 6 service starts at boot without problem.

-------------------------------------------------------
felix@elasticforum2:~ % doas service elasticsearch status
elasticsearch is running as pid 16306.
-------------------------------------------------------

I run the "rcorder /etc/rc.d/* /usr/local/etc/rc.d/*" command and it looks like the problem is that the Elasticsearch 6 service starts before the Clean /tmp service, but I’m not absolutely sure.

-------------------------------------------------------
/etc/rc.d/SERVERS
/usr/local/etc/rc.d/elasticsearch
/etc/rc.d/zfsd
/etc/rc.d/ntpdate
/etc/rc.d/rpcbind
/etc/rc.d/nisdomain
/etc/rc.d/ypserv
/etc/rc.d/ypxfrd
/etc/rc.d/ypupdated
/etc/rc.d/ypbind
/etc/rc.d/ypset
/etc/rc.d/yppasswdd
/etc/rc.d/ypldap
/etc/rc.d/wpa_supplicant
/etc/rc.d/virecover
/etc/rc.d/accounting
/etc/rc.d/nfsclient
/etc/rc.d/amd
/etc/rc.d/auditd
/etc/rc.d/auditdistd
/etc/rc.d/automountd
/etc/rc.d/automount
/etc/rc.d/autounmountd
/etc/rc.d/tmp
/etc/rc.d/cleartmp
-------------------------------------------------------

Thank you very much.
Comment 1 Juraj Lutter freebsd_committer freebsd_triage 2020-10-27 07:35:24 UTC
Thanks for the report, I (the maintainer) will review and work on this. It slipped through my finger somehow last time I did the elastic components review.
Comment 2 Nemesis11 2020-11-03 16:08:18 UTC
Thank you for reviewing this bug.

Just an update. I had this machine with FreeBSD 12.1 with the latest updates and I upgraded it to version 12.2 and I can confirm that this bug is still valid.

If I have the clear_tmp_enable="YES" service enabled in /etc/rc.conf, the elasticcsearch 6 service will fail to start.

After the boot, if I start the elasticsearch 6 service manually, it will start with no problem.

If I comment clear_tmp_enable="YES" service, the elasticsearch 6 service will start automatically at boot, with no problem.

I can replicate the problem in a second machine, that has more services then this one (nginx, mariadb104-server, etc), and it only happens with the elasticsearch 6 service.

I'm using the latest trimestral elasticsearch 6 package. Version elasticsearch6-6.8.12.

Output before and after the FreeBSD 12.2 update, with the clear_tmp_enable="YES" service enabled, at boot:


felix@elasticforum2:~ % doas uname -a
FreeBSD elasticforum2 12.1-RELEASE-p10 FreeBSD 12.1-RELEASE-p10 GENERIC  amd64
felix@elasticforum2:~ % doas service elasticsearch status
elasticsearch is not running.


felix@elasticforum2:~ % doas uname -a
FreeBSD elasticforum2 12.2-RELEASE FreeBSD 12.2-RELEASE r366954 GENERIC  amd64
felix@elasticforum2:~ % doas service elasticsearch status
elasticsearch is not running.


Thank you and Best Regards.
Comment 3 Juraj Lutter freebsd_committer freebsd_triage 2021-01-08 22:28:41 UTC
For the record, this bug applies to both elasticsearch6 and elasticsearch7.
It is, indeed, caused by wrong order in which the services are being started up.
Comment 4 commit-hook freebsd_committer freebsd_triage 2021-02-12 22:01:41 UTC
A commit references this bug:

Author: otis
Date: Fri Feb 12 22:01:23 UTC 2021
New revision: 565070
URL: https://svnweb.freebsd.org/changeset/ports/565070

Log:
  textproc/elasticsearch6: Fix rc order

  It has been found that elasticsearch is being started in wrong phase of rc
  sequence.

  This should bring it to more apropriate place where all services, including
  cleartmp, are started.

  PR:		236840
  Reported by:	Nemesis11 <felix@zwame.pt>
  Reviewed by:	osa (mentor), swills
  Approved by:	osa (mentor)
  MFH:		2021Q1
  Differential Revision:	https://reviews.freebsd.org/D27874

Changes:
  head/textproc/elasticsearch6/Makefile
  head/textproc/elasticsearch6/files/elasticsearch.in
Comment 5 commit-hook freebsd_committer freebsd_triage 2021-02-12 22:07:43 UTC
A commit references this bug:

Author: otis
Date: Fri Feb 12 22:06:59 UTC 2021
New revision: 565071
URL: https://svnweb.freebsd.org/changeset/ports/565071

Log:
  MFH: r565070

  textproc/elasticsearch6: Fix rc order

  It has been found that elasticsearch is being started in wrong phase of rc
  sequence.

  This should bring it to more apropriate place where all services, including
  cleartmp, are started.

  PR:		236840
  Reported by:	Nemesis11 <felix@zwame.pt>
  Reviewed by:	osa (mentor), swills
  Approved by:	osa (mentor)
  Differential Revision:	https://reviews.freebsd.org/D27874

Changes:
_U  branches/2021Q1/
  branches/2021Q1/textproc/elasticsearch6/Makefile
  branches/2021Q1/textproc/elasticsearch6/files/elasticsearch.in