Bug 225005

Summary: Mk/Uses/apache.mk: Rework 3rd party module building
Product: Ports & Packages Reporter: Miroslav Lachman <000.fbsd>
Component: Ports FrameworkAssignee: freebsd-apache (Nobody) <apache>
Status: Closed FIXED    
Severity: Affects Many People CC: brnrd, clement, eugen, joneum, ohauer, pi, ports-bugs
Priority: --- Keywords: needs-patch
Version: Latest   
Hardware: Any   
OS: Any   

Description Miroslav Lachman 2018-01-08 16:59:14 UTC
There are mess in 3rd party Apache modules from ports tree. They are messing with user edited config file httpd.conf. They use inconsistent way to manipulate this file.
As I already wrote https://lists.freebsd.org/pipermail/freebsd-ports/2017-October/110725.html some of them uses pre-deinstall, some post-deinstall.
But the main problem is they override users settings on each pkg upgrade.
If user decided to install AND manually enable some module (for example mod_xsendfile) this module should not modify httpd.conf on upgrade (deinstall) because it leaves Apache in broken state after each upgrade.

I think installation of module should not enable it it-self, the same way as services are not enabled in rc.conf on installation.
And if user did enable some module, deinstallation must not disable this module on upgrade.

There are tens of Apache modules in the ports tree and I think all of them must use the same way of install / deinstall.

For example - mod_php71 is enabled in httpd.conf on install, removed on deinstall.
But mod_xsendfile is added to httpd.conf commented, user uncomment this line and then pkg upgrade (deinstall phase) will remove this line, install will add it back commented out. It's a real pain to maintain Apache + modules if each are behaving different.

The root cause partialy lies in ports/Mk/bsd.apache.mk which is responsible for manipulating httpd.conf but some ports are using own way to edit httpd.conf.


There are two ways (at least) to handle this properly:

1) do not manipulate httpd.conf in any way and just print pkg-message with information what should be added in to httpd.conf manually be user (or we can show the right command "apxs -e -a -n php5 libphp5.so" or "apxs -e -a -n xsendfile 
mod_xsendfile.so" so the user can simply run one command to enable the module)

2) install *.sample file for each module in to apache24/modules.d/ and if user decides to enable module (copy or rename mod_xsendfile.conf.sample to mod_xsendfile.conf) then pkg upgrade / pkg delete should not touch this conf file


How-to-reproduce:

1) install Apache 2.4
2) make a copy of directory apache24
  # cp -Rp apache24 apache24.1

3) # pkg install ap24-mod_xsendfile

4) # diff -r -u apache24.1 apache24
+#LoadModule xsendfile_module   libexec/apache24/mod_xsendfile.so

5) Manually enable this module
# sed -i '' -E 's/#(LoadModule.*mod_xsendfile.so)/\1/' apache24/httpd.conf

6) # diff -r -u apache24.1 apache24
+LoadModule xsendfile_module   libexec/apache24/mod_xsendfile.so

7) Try to upgrade it:
# pkg upgrade -f ap24-mod_xsendfile

8) Here comes the chaos, config file does not have xsendfile at all
# diff -r -u apache24.1 apache24
# grep xsendfile apache24/httpd.conf

If you use pkg delete + pkg install instead of pkg upgrade -f, then you will have commented LoadModule (disabled xsendfile) in httpd.conf


The second example with mod_php is different, mod_php is enabled again even if user wants module installed (and then upgraded) but keep module disabled:

1) # pkg install mod_php71
2) # diff -r -u apache24.1 apache24
+LoadModule php7_module        libexec/apache24/libphp7.so

3) Manually disable the module
# sed -i '' -E 's/#?(LoadModule.*libphp7.so)/#\1/' apache24/httpd.conf

# grep php7 apache24/httpd.conf
#LoadModule php7_module        libexec/apache24/libphp7.s

4) run upgrade after some time
# pkg upgrade -f mod_php71

5) module is accidentally enabled again
# grep php7 apache24/httpd.conf
LoadModule php7_module        libexec/apache24/libphp7.so

6) and removed in deinstall
# pkg delete mod_php71

7) # grep php7 apache24/httpd.conf


If there is written policy on how to (not) handle Apache module install / upgrade / deinstall then it should be strictly followed by all modules from ports tree.

If policy does not exists then I think it should be created and followed be all maintainers and committers.
Comment 1 Eugene Grosbein freebsd_committer freebsd_triage 2018-01-08 18:10:08 UTC
It seems we need clarified policy for maintainters of Apache modules that should describe how to deal with modules on deinstallation/upgrade to not break things and not ruin user-modifien configuration files.

Such policy needs to be consistent with code of ports/Mk/bsd.apache.mk.
Do we already have such documentation for maintainers?
Comment 2 Miroslav Lachman 2018-01-08 18:33:33 UTC
The problem is that current code in ports/Mk/bsd.apache.mk is buggy - as I demonstrated above, mod_xsendfile does not use any own code for handling httpd.conf modification, it uses just these variables in Makefile:

USE_APACHE=     22+
AP_FAST_BUILD=  yes
AP_GENPLIST=    yes

Everything else is in bsd.apache.mk and as shown above it will remove my modified settings on upgrade.
Comment 3 Eugene Grosbein freebsd_committer freebsd_triage 2018-01-13 08:03:26 UTC
(In reply to Miroslav Lachman from comment #2)

Can you propose a fix to that code?
Comment 4 Miroslav Lachman 2018-01-13 11:59:22 UTC
(In reply to Eugene Grosbein from comment #3)
Sure, I can try to write something but I don't want to spend time writing something which will not be considered useful and commitable. So can we first choose what the right method should be?

1) do not touch httpd.conf at all and just print pkg-message instructions to users
  or
2) create module_name.conf.sample in apache24/modules.d/

Which one is preferred way?
Comment 5 Eugene Grosbein freebsd_committer freebsd_triage 2018-01-13 12:32:33 UTC
(In reply to Miroslav Lachman from comment #4)

The port www/apache24 already installs *.sample configuration files consistently with Porters Handbook, so that should be just fine.

Some additional hints in pkg-messages won't hurt too, so you can do both ways, if you wish, but *.sample's are needed anyway.
Comment 6 Miroslav Lachman 2018-01-13 12:59:11 UTC
(In reply to Eugene Grosbein from comment #5)
OK, I'll try to code something in the next week (if time permits) 
Thank you.
Comment 7 Jochen Neumeister freebsd_committer freebsd_triage 2018-02-25 13:54:06 UTC
please note: https://reviews.freebsd.org/D12308

Here we are currently in the last final tests
Comment 8 Bernard Spil freebsd_committer freebsd_triage 2018-03-18 20:48:11 UTC
I've heard discussions of moving module configuration to something like modules.d to cope with these changes. With the migration to USES behind us, we could start working on this. I like the idea of making this more modular. Any tips/hints?
Comment 9 Miroslav Lachman 2018-03-19 13:28:03 UTC
(In reply to Bernard Spil from comment #8)
What is the current state? Are there consensus to user module_name.conf.sample in apache24/modules.d/? If yes, then I would like to see all apache modules converted to install sample file instead of touching httpd.conf unconditionally.

I was on vacation and then busy with $WORK so I didn't touch this PR for a while.
Do we have something in ports/Mk/ to handle installing sample files for Apache modules or do we need to code something?
Comment 10 Bernard Spil freebsd_committer freebsd_triage 2018-03-19 21:07:31 UTC
(In reply to Miroslav Lachman from comment #9)

> would like to see all apache modules converted

I assume you refer modules not from the www/apache24 only?

I believe there's some form of consensus on using modules.d, I for one would be happy to support that. Beware that a number of modules are on the verge of having to be deleted, as they are 2.2 only. See https://wiki.freebsd.org/Apache#Apache_2.2

Additional change is that we'll need to add an Include modules.d.

Personally, I like to set Global Defines outside of the config and include that for paths I want to use SSLDir, LogDir, ModConfDir ...
Comment 11 Bernard Spil freebsd_committer freebsd_triage 2018-03-19 21:31:48 UTC
Just read your post to freebsd-ports@ from 2017-Nov.

I can see us adding mechanism to Mk/Uses/apache.mk to build modules.

Thinking up to now...

Add arg module to USES= apache to signal that we're building a module. This would also allow us to document more closely.

AP_FAST_BUILD to be renamed and used as auto-build and install mechanism. It is not FAST, just an easier template for plist generation and installation.
Install is using @sample to install into modules.d and echo message on enabling.
modules.d files can also use @sample template as they should adhere to 

> # LoadModule modname
> <IfModule modname>
> ModuleConf here
> </IfModule>

to force it not to auto-load include not fail after installation.
The whole ap-gen-plist target could possibly disappear by using PLIST_FILES instead.

mod_phpNN is a difficult one altogether, it should be ap24-mod_phpNN (or ap22-mod_phpNN) but I guess that'll break lang/phpNN's Makefile.

Bernard (with hat apache@)
Comment 12 commit-hook freebsd_committer freebsd_triage 2018-03-24 22:01:57 UTC
A commit references this bug:

Author: brnrd
Date: Sat Mar 24 22:01:35 UTC 2018
New revision: 465488
URL: https://svnweb.freebsd.org/changeset/ports/465488

Log:
  www/mod_md-devel: Add missing files

   - Pulled the trigger early on 465487

  PR:		225005

Changes:
  head/www/mod_md-devel/files/999_mod_md.conf.in
  head/www/mod_md-devel/files/patch-src_md__crypt.c
  head/www/mod_md-devel/files/pkg-message.in
  head/www/mod_md-devel/pkg-message
  head/www/mod_md-devel/pkg-plist
Comment 13 Eugene Grosbein freebsd_committer freebsd_triage 2018-04-28 19:59:25 UTC
Why is this closed without a commentary?
Comment 14 Jochen Neumeister freebsd_committer freebsd_triage 2018-04-28 20:07:11 UTC
(In reply to Eugene Grosbein from comment #13)

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225005#c12
Comment 15 Eugene Grosbein freebsd_committer freebsd_triage 2018-04-28 20:42:18 UTC
(In reply to Jochen Neumeister from comment #14)

I fact, that commit r465488 has no connection with this PR. Are you sure you understand the matter?
Comment 16 Kurt Jaeger freebsd_committer freebsd_triage 2018-04-29 04:26:31 UTC
That bug was closed, but it should remain open as the issue is no fixed yet.
Comment 17 Bernard Spil freebsd_committer freebsd_triage 2018-05-02 19:37:31 UTC
I've added support to mod_md-devel and mod_http2-devel.

Numbering the .conf files was a complete guess. How do we go about numbering these properly?

If we figure that out, we can go and add support to the ports.
Comment 18 Jochen Neumeister freebsd_committer freebsd_triage 2020-03-14 21:08:09 UTC
Now that nothing has happened here for two years, I'm closing down. If the problem still exists, please open with a current error description.