Bug 218844
I've been working on something similar, let me know if you need any help testing or with the rc.d script. (In reply to Reshad from comment #1) reshad yes please lets share notes. I'm on freenode in #couchdb-dev channel most of the time. Created attachment 182233 [details]
v0.2 patch builds, rc.d start/stop, still needs refinement
Hey,
Went through your port and merged it whit what I had.
The port seems to work now, but still needs some polish:
1. rc script to start and stop works, but is a hack.
The couchdb command(shell script) daemonises without giving us a pid.
Alos logging (pipe stdout and stderr to file) is a hack.
2. Configs used are in /usr/local/lib/couchdb/etc.
Need to find a way to have couchdb take configuration files in /usr/local/etc/couchdb
3. pkg plist/files installed need to be trimmed down
Possibly re-writing the script to start couchdb which takes some parameters such as config files, log location, etc would be the way to go.
Created attachment 182288 [details]
v0.3 patch builds small refinement to rc.d stop start logic needed
Logs to /var/log/couchdb/couchdb.log
Reads config files from /usr/local/etc/couchdb/*
TODO:
1. PID file
2. reduce pkg-plist (installed files)
Created attachment 182420 [details]
v0.4 Fix permissions issue, Fauxton self test passes, more refined rc.d script
This patch fixes some permissions issue with the previous version.
The rc.d script is also cleaner and should handle better.
Fauxton self test for DB passes.
TODO:
- Stop should stop all processes running as couchdb (currently, it only kills the beam process, which is usually good enough, but sometimes it does not die)
- Cleanup make file a bit.
- Generate pidfile (nice to have).
Created attachment 182429 [details]
v0.5 Fix rc.d script, cleanup Makefile, set permissions for /var
New patch updates the old one with these changes:
- Cleanup echos and testing comments in Makefile.
- rc.conf erl_flags configurable as an rc_var
- rc.d start daemon now uses -f to pipe unwanted output to /dev/null
- fix bug in rc.d stop (status_quiet)
- fix make package warning about etc/rc.d/couchdb declared twice
- set permissions to 740 for /var/{log,db,run}/couchdb
Issues
- pidfile still not generated
The pidfile works with the current revision (v0.5) @Dave, let me know if there is any other change you would like to make. I have this running in a testing environment at the moment and its working fine so far. I think this is ready to be committed to the tree. Created attachment 182475 [details]
v6 uses WTFPL License just patched in ports tree
Updated with WTFPL
Created attachment 182543 [details]
v7 config file cleanup + portlint
Cleanup the etc/couchdb dir by putting sample files in the lib/couchdb/etc/ directory.
adhere to portlint -A
Created attachment 182800 [details]
v8 update versions in plist
Updated erts and lib depend versions in plist.
This can be a problem in the future.
Hi,
I'm trying to install your port (version v8) but `make install` fails; do you have any idea on how to fix this?
My test environment is a fresh FreeBSD 11 installation running on a virtual machine.
Thanks
Installing CouchDB into rel/couchdb/ ...
==> rel (generate)
WARN: 'generate' command does not apply to directory /usr/ports/databases/couchdb2/work/apache-couchdb-2.0.0
... done
You can now copy the rel/couchdb directory anywhere on your system.
Start CouchDB with ./bin/couchdb from within that directory.
gmake[1]: Leaving directory '/usr/ports/databases/couchdb2/work/apache-couchdb-2.0.0'
===> Staging for couchdb2-2.0.0
===> couchdb2-2.0.0 depends on package: erlang>=19 - found
===> Generating temporary packing list
===> Creating groups.
===> Creating users
install -m 0644 /usr/ports/databases/couchdb2/work/apache-couchdb-2.0.0/share/docs/pdf/CouchDB.pdf /usr/ports/databases/couchdb2/work/stage/usr/local/share/doc/couchdb2
install -m 444 /usr/ports/databases/couchdb2/work/apache-couchdb-2.0.0/share/docs/man/apachecouchdb.1 /usr/ports/databases/couchdb2/work/stage/usr/local/man/man1
/bin/mv /usr/ports/databases/couchdb2/work/apache-couchdb-2.0.0/rel/couchdb/share /usr/ports/databases/couchdb2/work/stage/usr/local/share/couchdb
/bin/mv /usr/ports/databases/couchdb2/work/apache-couchdb-2.0.0/rel/couchdb/etc/vm.args /usr/ports/databases/couchdb2/work/apache-couchdb-2.0.0/rel/couchdb/etc/vm.args.sample #/usr/ports/databases/couchdb2/work/stage/usr/local/etc/couchdb/vm.args.sample
/bin/mv /usr/ports/databases/couchdb2/work/apache-couchdb-2.0.0/rel/couchdb/etc/local.ini /usr/ports/databases/couchdb2/work/apache-couchdb-2.0.0/rel/couchdb/etc/local.ini.sample #/usr/ports/databases/couchdb2/work/stage/usr/local/etc/couchdb/local.ini.sample
cp -f -R -p /usr/ports/databases/couchdb2/work/apache-couchdb-2.0.0/rel/couchdb /usr/ports/databases/couchdb2/work/stage/usr/local/lib/
====> Compressing man pages (compress-man)
===> Staging rc.d startup script(s)
===> Installing for couchdb2-2.0.0
===> Checking if couchdb2 already installed
===> Registering installation for couchdb2-2.0.0
pkg-static: Unable to access file /usr/ports/databases/couchdb2/work/stage/usr/local/root@FreeBSD12-20-33:/usr/ports/databases/couchdb2 # cat pkg-plist:No such file or directory
pkg-static: duplicate file listing: /usr/local/lib/couchdb/etc/local.ini.sample, ignoring
pkg-static: duplicate file listing: /usr/local/lib/couchdb/etc/vm.args.sample, ignoring
*** Error code 74
Stop.
make: stopped in /usr/ports/databases/couchdb2
Created attachment 183273 [details]
v9 update versions in plist
update versions for erlang-19.3.5,3
(In reply to sgironella from comment #11) Hi, Sorry v8 had an error in the pkg-plist file. This is fixed in v9. v9 also takes into account an updated version of erlang committed 2 days back, it may not be in packages yet. Let me know how it goes, I have only tested with 12-CURRENT and limited testing on 10.3-RELEASE. (In reply to Reshad Patuck from comment #13) Hi, now it's running fine. Thanks! Comment on attachment 182041 [details]
v0.1 patch builds successfully but needs rc.d script and BSD-friendly file locations
Obsoleted by Reshads patch (Discussed via IRC)
Bug 212846 is not blocked by this, it in fact is a proposed change that would deprecate/supersede this issue. Opinions aside, one of these issues needs to be closed in favour of the other, depending on whether the existing port will cross a major boundary from 1.x -> 2.x, or a new port (databases/couchdb2) is created. With that in mind, this is existing maintainer related (request feedback accordingly) @rap aside from the OTP 20.0 vs OTP 19.x clashes discussed on IRC there are a few changes needed for pkg-plist yet: - /usr/local/lib/* should be read only in general and root:wheel (default) groups - 0755 for dirs, 0644 for files (defaults) - sensitive files should be couchdb:couchdb 0600 or root:couchdb 0660 (local.ini, vm.args, sys.config) - default.ini should be 0644 root:wheel as its part of the distribution - directories should be root:wheel 0755 apart from /var/*/couchdb which should be root:couchdb 0770 to prevent information disclosure - .o, .a .c intermediate files should be removed - the port should install to /usr/local/lib/couchdb2 so that people who wish to migrate from v1.x to v2.x can have them concurrently installed, at least via `pkg install -f couchdb couchdb2` - this would mean you can use prefixes instead of custom locations in the pkg-plist: DATADIR=/usr/local/share/couchdb2 DOCSDIR=/usr/local/share/doc/couchdb2 EXAMPLESDIR=/usr/local/share/examples/couchdb2 WWWDIR=/usr/local/www/couchdb2 ETCDIR=/usr/local/etc/couchdb2 using https://pkg.skunkwerks.at/poudriere/build.html?mastername=11_amd64-default&build=2017-07-11_09h40m41s as reference for errors, which is your latest patch + an updated erts version spliced in. Created attachment 184308 [details]
v10 Multiple updates to plist, rc.d script and Makefile
Hi,
I have updated the port with the following changes:
- Changed dependency on 'erlang' to 'erlang-runtime19'.
- Change 'couchdb' to 'couchdb2' allowing for both the version 1 port and version 2 port to coexist if needed.
- Cleanup source/staging (*.a, *.c and *.o) files.
- Use wait_for_pids in the couchdb2 rc stop script to wait till the couchdb pid is dead.
- Change permissions of the default config files to couchdb:couchdb 0660.
- Bumped erts version to 8.3.5.1.
- Changed perms in DATADIR (www) from 0755 to 0744.
- Changed perms of var dirs to couchdb:couchdb 0770.
@dch
I have NOT changed the perms in /usr/local/lib/*.
Most of the files are already root:wheel 0644.
The last time I tried changing the perms (some months back) in this dir couch refused to start.
That is why I am using a 'cp' to copy them over.
Do let me know if there are any other changes you would like.
Best,
Reshad
Created attachment 184911 [details]
v11 go back to erlang insted of erlang-runtime19
Changed dependency from 'erlang-runtime19' back to 'erlang'.
erlang-runtime19 installs to its own directory which needs to be in path.
@Dave, if you have a way to point couchdb to a particular path for erlang then we can use erlang-runtime19.
I don't see any ./configure options to do this.
Pending QA confirmation (this applies to each newly proposed patch/attachment) Created attachment 184939 [details]
v12 updated erts version
Bumped erts version from 'erts-8.3.5.1' in pkg-plist to 'erts-8.3.5.2'
Created attachment 184940 [details]
v12 - poudriere log
poudriere log for v12 patch
Created attachment 185036 [details]
v13 - patch - update to couchdb 2.1.0
Updates the patch to version 2.1.0 of couchdb
Files changed/updated since last patch:
- Makefile (new version + removed line for pdf docs that do not exist in new version)
- distinfo
- files/patch-* (slight changes to patch files, just moving lines up or down as needed)
- pkg-message (point to docs for 2.1.0 instead of 2.0.0)
- pkg-plist (updated)
Created attachment 185037 [details]
v13 - poudriere log
* Uses ssl is being used. In order to link ssl to the correct branch the variables OPENSSLBASE, OPENSSLDIR, OPENSSLLIB, OPENSSLINC, OPENSSLRPATH maybe needed. Please check how the port is linking to ssl and adapt to respect the variables required. * The rc script has many hardcoded paths this should be changed, for example %%PREFIX%% for /usr/local. Also, consider adding more configuration variables for default.ini/local.ini. * Why are executables/docs/includes/misc? being installed in %%PREFIX%%/lib/couchdb2? This should probably be changed to the appropriate directories. If for whatever reason the software can't run in a different hierarchy, which I doubt, it should probably have its home directory somewhere like /usr/local/couchdb2. Created attachment 186144 [details]
v14 - patch - move files to %%PREFIX%%/couchdb2
Sorry for the late update, Ive been unavailable for the last month or so.
As per @Richard's recommendations I have:
- Moved files from %%PREFIX%%/lib/couchdb2 to %%PREFIX%%/couchdb2.
- Removed hardcoded paths in the rc script in favour of templates
@Richard
I have tried to seperate the couchdb files out so that it runs from the appropriate directories, but have not gotten it to run.
Further when speaking with the couchdb devs about this they recommend it stays altogether.
I am not sure how this port is linking to ssl. Any help here would be appreciated
Created attachment 186145 [details]
v14 - poudriere log
Created attachment 193529 [details]
v15 - patch - couchdb 2.1.1 port
Here are some improvements
1. installs couchdb version 2.1.1
2. installs into /usr/local/libexec/couchdb2
3. builds most of pkg-plist dynamically so it works with minor OPT version changes and other libs version changes like mochiweb
4. adds some configuration options
Created attachment 193540 [details]
v16 - patch couchdb 2.1.1 port - portlint happy
same as v15 but portlint is happy
Is a complete Port of CouchDB 2.1.1 in the making? This needs a bit more cleanup, which can be done by the committer taking it on. Many bits are in the wrong order, See Chapter 15. Order of Variables in Port Makefiles, https://www.freebsd.org/doc/en/books/porters-handbook/porting-order.html from the porter's handbook. In BUILD_DEPENDS the ${LOCALBASE}/bin/ should be removed. PLIST should not be changed from its default, and all the `find >> ${PLIST}` should be changed to `find >> ${TMPPLIST}`. bsd.port.options.mk should not be included, all the following .if should use options helpers instead. http://www.freebsd.org/doc/en/books/porters-handbook/makefile-options.html pre-install and do-install should be merged. Files MUST NOT be installed with MV or CP, they MUST be installed with one of the INSTALL_* macro or using the COPYTREE_* macros. https://www.freebsd.org/doc/en/books/porters-handbook/install.html The man page MUST always be installed, and not be dependant on a DOCS option. Oh, and in pkg-plist, `@dir %%ETCDIR%%` should be removed. (Or better, the plist should be generated using make makeplist, and not be made dynamic.) Take. I'll work on this in the next few days. Created attachment 194254 [details]
patch-v17
New version, hopefully covering all comments.
Testbuilds on 12a, 11.2a, 11.1a, 10.4i are fine.
Take, in coordination with ler. v17 looks close enough to be almost ready. Is there a good reason to support both erlang runtimes ? This causes issues with the pkg-plist. It would be much easier if we stay with erlang-runtime20, only. Created attachment 194257 [details]
patch-v18
Tested most options. erlang19 is still broken because the pkg-plist is wrong if 19 is selected.
I'm unsure about conflicts with couchdb1 -- any hints ?
*** Bug 212846 has been marked as a duplicate of this bug. *** Created attachment 194259 [details]
patch-v19
So, that should be the final version.
some comments; super delighted to see this moving again & LGTM.
- there is no direct "copy the files" migration from couchdb 1.x -> 2.x so ideally you'd be able to run both versions side by side. It looks like this is sorted - nice!
- I see no reason at all to build against OTP 20 as it's not supported upstream, and the build produces some very reasonable errors in the multipart mime parser, and http session handling:
/tmp/usr/ports/databases/couchdb2/work/apache-couchdb-2.1.1/src/mochiweb/src/mochiweb_multipart.erl:41: Warning: call to crypto:rand_bytes/1 will fail, since it was removed in 20.0; use crypto:strong_rand_bytes/1
/tmp/usr/ports/databases/couchdb2/work/apache-couchdb-2.1.1/src/mochiweb/src/mochiweb_session.erl:125: Warning: call to crypto:rand_bytes/1 will fail, since it was removed in 20.0; use crypto:strong_rand_bytes/1
/tmp/usr/ports/databases/couchdb2/work/apache-couchdb-2.1.1/src/couch_epi/src/couch_epi_codegen.erl:78: Warning: erl_scan:token_info/2: removed in 19.0; use erl_scan:{category,column,line,location,symbol,text}/1 instead
- fauxton doesn't need to be a build time option, it's only required if you intend to download and build the nodejs UI from scratch, and this is already provided by couchdb release tarball. Any production couch instance should have fauxton ui available.
- curl is not a production requirement, only to run the javascript test suite. the couchjs binary should not have this included in production as it might allow a js function to use it to escape its sandbox
The daemon doesn't start for me, looks like too aggressive pruning of executable permissions. This should fix it:
chmod +x /usr/local/libexec/couchdb2/erts-*/bin/* \
/usr/local/libexec/couchdb2/lib/*/priv/bin/* \
/usr/local/libexec/couchdb2/bin/*
But there are a few others inside ERTS as well I may have missed.
(In reply to Dave Cottlehuber from comment #40) Thanks for your comments and testing. I've tested the startup now and find that it looks like the database is not starting at all, as the rc.d script looks for pgrep beam but that process is not started. Maybe it's because there's no default.ini file ? Those are only installed if OTP20 is selected (?). It seems there's still some open issue somehow ? Building it with OTP20 allows the beam* processes to run, but: if I then proceed as explained in the pkg-message, the curl calls work. Proceeding as described in http://docs.couchdb.org/en/2.1.1/install/setup.html#single-node-setup to the URL http://127.0.0.1:5984/_utils#setup it works until a point and some errors pop up. I have no idea about couchdb2, so if anyone has more experience... ? "make install" is already broken again. At least for OTP20. Same will happen with OTP19. Cause is the *static* pkg plist. Couchdb2 builds as an erlang application package which self-contains the complete OTP which is taken from the currently installed erlang-runtime20-20.x.y.z. For example pkg-plist wants to copy files from "erts-9.3.2" but OPT20 is already at "erts-9.3.3". So the port will break with each minor OTP update. Ok, now I understand your comment about the dynamic plist. We still have issues with OTP20 not working completly and OTP19 not working at all, no ? I'm unable to create users in the OTP20 version, does this work for anybody else? curl -X PUT http://admin:passwd@localhost:5984/_users/org.couchdb.user:beastie -H "Accept: application/json" -H "Content-Type: application/json" -d '{"name": "beastie", "password": "whatiscoredoingaboutit", "roles": [], "type": "user"}' BTW wearing my "upstream" hat we should be fine with OTP20 and the 3 small patches, but it's a bit odd nobody else has mentioned/noticed this already; I'll not get to submitting them until monday earliest though as I borked my ports computer. Neither OTP19 nor OTP20 works.
As already mentioned OTP20 fails at "make install" with file not found erros. pkg-plist points to "%%OTP20%%libexec/couchdb2/erts-9.3.2/" but the "make" process copies the erts directory "from /usr/local/lib/erlang20/erts-9.3.3/" which is installed when using the latest ports tree with svn update. Btw. "pkg install erlang-runtime20" installs erts currently in "/usr/local/lib/erlang20/erts-9.3/"
So I think it is impossible to build an always working port for couchdb app with self-contained erts when using a static pkg-plist.
For OTP19 "make install" works, because pkg-plist still matches the erlang runtime system in "/usr/local/lib/erlang19/erts-8.3.5.4/".
But you can't start it (service start couchdb2) because none of the installed files have +x permission. Reason is the use of ${COPYTREE_SHARE} instead of ${CP}.
@Dave: Yes, I can create users with OTP20 without any problems. (provided that I've got a working couchdb2 installation like from patch v16)
There is an option "excl_lib" in erlang reltool (http://erlang.org/doc/man/reltool.html), but it's marked experimental. Setting excl_lib to otp_root in work/.../rel/reltool.config prevents embedding of OTP. Also some modifications in bin/couchdb are needed to get it to work. So this would be a possibility to stick on a static plist, because OTP hast not to be copied at "make install". But this feature is experimental and not the way couchdb2 should be built. At most this could be offered as experimental feature in "make config" which could be selected to save some disk space. Request feedback from MAINTAINER listed in latest patch. If someone else wants to progress this, please provide an updated patch addressing any remaining issues (and updating the MAINTAINER line if needed) Comment on attachment 194259 [details] patch-v19 I moved discussion to https://reviews.freebsd.org/D16819 with proposed patch that complies with expected permissions, file locations, allows running couch 1.7.x and 2.x concurrently to ease migration, and depends on the OTP files from lang/erlang-runtime* to make port maintenance easier. It's missing the ability to split out the HTML docs, as previous ports removed fauxton (IMO required),and also some of the javascripts needed for views. I'll look at adding them back, and probably removing OTP19 support as I see no real-world use case for it. comments welcomed & many thanks for everybody's contributions. A commit references this bug: Author: dch Date: Fri Sep 21 15:21:02 UTC 2018 New revision: 480279 URL: https://svnweb.freebsd.org/changeset/ports/480279 Log: databases/couchdb2: new port via PR#218844 Summary: JSON document database with HTTP API & scalable multi-master sync A number of people provided signficant assistance while developing this port. Thank-you for your perseverance and assistance. PR: 218844 Submitted by: dch Reviewed by: Reshad Patuck, Richard Gallamore, Tobias Herre, mat, ler, pi Approved by: jrm (mentor) Differential Revision: https://reviews.freebsd.org/D16819 Changes: head/databases/Makefile head/databases/couchdb2/ head/databases/couchdb2/Makefile head/databases/couchdb2/distinfo head/databases/couchdb2/files/ head/databases/couchdb2/files/couchdb2.in head/databases/couchdb2/files/patch-configure head/databases/couchdb2/files/patch-rel_reltool.config head/databases/couchdb2/files/patch-src_ibrowse_src_ibrowse__http__client.erl head/databases/couchdb2/files/pkg-message.in head/databases/couchdb2/pkg-descr head/databases/couchdb2/pkg-plist |
Created attachment 182041 [details] v0.1 patch builds successfully but needs rc.d script and BSD-friendly file locations As 2.0.0 adds clustering support and is not a drop-in replacement it seems very appropriate to add it as a new port for the time being, and on the next lang/erlang update (which will require patching 1.6.x) we could consider providing an UPDATING note with migration details.