Bug 198073 - [NEW PORT] databases/influxdb: Open-source distributed timeseries database
Summary: [NEW PORT] databases/influxdb: Open-source distributed timeseries database
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Steve Wills
URL:
Keywords: needs-qa, patch
Depends on:
Blocks:
 
Reported: 2015-02-27 14:56 UTC by cheffo
Modified: 2015-06-26 13:22 UTC (History)
5 users (show)

See Also:


Attachments
influxdb.shar file (7.96 KB, text/plain)
2015-02-27 14:56 UTC, cheffo
no flags Details
poudriere testport output (18.14 KB, text/plain)
2015-02-27 14:57 UTC, cheffo
no flags Details
poudriere testport portlint output (91 bytes, text/plain)
2015-02-27 14:57 UTC, cheffo
no flags Details
influxdb090r7.shar (9.82 KB, text/plain)
2015-03-04 16:41 UTC, cheffo
no flags Details
patch to add influxd user in ports tree (601 bytes, patch)
2015-03-04 16:42 UTC, cheffo
no flags Details | Diff
patch to add influxd group in ports tree (234 bytes, patch)
2015-03-04 16:42 UTC, cheffo
no flags Details | Diff
poudriere testport output (18.82 KB, text/plain)
2015-03-04 16:43 UTC, cheffo
no flags Details
Influxdb v0.9.0 - release (14.14 KB, text/plain)
2015-06-23 15:03 UTC, cheffo
no flags Details
poudriere testport output (19.61 KB, text/plain)
2015-06-23 15:04 UTC, cheffo
no flags Details
Updated version of the port which follows and uses new model for github. (11.59 KB, text/plain)
2015-06-24 08:52 UTC, cheffo
no flags Details
poudriere testport output attached. (19.76 KB, text/plain)
2015-06-24 08:53 UTC, cheffo
no flags Details
Updated patch (15.39 KB, patch)
2015-06-26 00:48 UTC, Steve Wills
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description cheffo 2015-02-27 14:56:28 UTC
Created attachment 153579 [details]
influxdb.shar file

New port:

An open-source, distributed, time series database with no external dependencies.
InfluxDB is the new home for all of your metrics, events, and analytics.

InfluxDB is a time series, metrics, and analytics database. It'written in Go and
has no external dependencies. That means once you install it there's nothing
else to manage (like Redis, ZooKeeper, HBase, or whatever).

InfluxDB is targeted at use cases for DevOps, metrics, sensor data, and
real-time analytics. It arose from our need for a database like this on more
than a few previous products we' built. You can read more about our jurney from
SaaS application to open source time series database.

WWW: http://influxdb.com
Comment 1 cheffo 2015-02-27 14:57:09 UTC
Created attachment 153580 [details]
poudriere testport output
Comment 2 cheffo 2015-02-27 14:57:39 UTC
Created attachment 153581 [details]
poudriere testport portlint output
Comment 3 Dave Cottlehuber freebsd_committer 2015-02-28 01:18:23 UTC
Thanks for the huge amount of work in this port, the rewriting between 0.8 to 0.9 is ... extensive.


I tidied up the port structure a bit, see https://github.com/skunkwerks/freebsd-ports/commit/4c9fa11 in particular:

- trying to get the daemon name (influxd) and the port name (influxdb) lined up again.
- the shas in distinfo didn't match what the scripts expect, so those are fixed too. I think this is go version drift in action.
- I added a dedicated influxdb user account for cleaner security
- used /usr/sbin/daemon to manage influxd itself. While influxdb *may* provide this functionality now (I didn't look closely enough), daemon is more trusted code than a relatively new go service.

Unfortunately I've not gotten this all working completely yet, but it seems subjectively cleaner to me this way. I hope its useful. portlint & initial building on 10.1 amd64.

```
❯❯❯ sudo make deinstall clean fetch makesum stage stage-qa check-orphans check-plist package install

===>  Deinstalling for influxd
===>   Deinstalling influxd-0.9.0.rc5
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
        influxd-0.9.0.rc5

The operation will free 20 MiB.
[1/1] Deinstalling influxd-0.9.0.rc5...
[1/1] Deleting files for influxd-0.9.0.rc5: 100%
===>  Cleaning for influxd-0.9.0.rc5
===>  License MIT accepted by the user
===>   influxd-0.9.0.rc5 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by influxd-0.9.0.rc5 for building
===>  License MIT accepted by the user
===>   influxd-0.9.0.rc5 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by influxd-0.9.0.rc5 for building
===>  License MIT accepted by the user
===>   influxd-0.9.0.rc5 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by influxd-0.9.0.rc5 for building
===>  Extracting for influxd-0.9.0.rc5
=> SHA256 Checksum OK for influxd-0.9.0.rc5.tar.gz.
=> SHA256 Checksum OK for BurntSushi-toml-443a628.tar.gz.
=> SHA256 Checksum OK for bmizerany-pat-b8a3500.tar.gz.
=> SHA256 Checksum OK for boltdb-bolt-fba9b58.tar.gz.
=> SHA256 Checksum OK for kimor79-gollectd-1d0fc88.tar.gz.
=> SHA256 Checksum OK for peterh-liner-29f6a64.tar.gz.
=> SHA256 Checksum OK for rakyll-statik-4a16c83.tar.gz.
=> SHA256 Checksum OK for c3294304d93f48a37d3bed1d382882a9c2989f99.tar.gz.
=> SHA256 Checksum OK for 1351f936d976c60a0a48d728281922cf63eafb8d.tar.gz.
=> SHA256 Checksum OK for 35bc42037350f0078e3c974c6ea690f1926603ab.tar.gz.
===>  Patching for influxd-0.9.0.rc5
===>   influxd-0.9.0.rc5 depends on package: go>=1.4.0 - found
===>   influxd-0.9.0.rc5 depends on executable: hg - found
===>   influxd-0.9.0.rc5 depends on executable: git - found
===>   influxd-0.9.0.rc5 depends on file: /usr/local/bin/go - found
===>  Configuring for influxd-0.9.0.rc5
===>  Building for influxd-0.9.0.rc5
===>  Staging for influxd-0.9.0.rc5
===>   Generating temporary packing list
/bin/mkdir -p /usr/ports/skunkwerks/influxdb/work/stage/usr/local/etc/influxd  /usr/ports/skunkwerks/influxdb/work/stage/var/db/influxd/  /usr/ports/skunkwerks/influxdb/work/stage/var/run/influxd/  /usr/ports/skunkwerks/influxdb/work/stage/var/log/influxd/
install   -o root -g wheel -m 555 /usr/ports/skunkwerks/influxdb/work/bin/influxd /usr/ports/skunkwerks/influxdb/work/stage/usr/local/bin/influxd
install   -o root -g wheel -m 555 /usr/ports/skunkwerks/influxdb/work/bin/influx /usr/ports/skunkwerks/influxdb/work/stage/usr/local/bin/influx
install  -o root -g wheel -m 0644  /usr/ports/skunkwerks/influxdb/work/src/github.com/influxdb/influxdb/etc/config.sample.toml  /usr/ports/skunkwerks/influxdb/work/stage/usr/local/etc/influxd/influxd.conf.sample
====> Compressing man pages (compress-man)
===> Staging rc.d startup script(s)
====> Running Q/A tests (stage-qa)
====> Running Q/A tests (stage-qa)
====> Checking for pkg-plist issues (check-plist)
===> Parsing plist
===> Checking for items in STAGEDIR missing from pkg-plist
===> Checking for items in pkg-plist which are not in STAGEDIR
===> No pkg-plist issues found (check-plist)
===>  Building package for influxd-0.9.0.rc5
===>  Installing for influxd-0.9.0.rc5
===>  Checking if influxd already installed
===>   Registering installation for influxd-0.9.0.rc5
Installing influxd-0.9.0.rc5...

❯❯❯portlint

WARN: Makefile: possible use of absolute pathname "/var/db/${PORTNAME}/".
WARN: Makefile: possible use of absolute pathname "/var/log/${PORTNAME}...".
WARN: Makefile: possible use of absolute pathname "/var/run/${PORTNAME}...".
FATAL: Makefile: PORTVERSION looks illegal. You should modify "0.9.0.rc5".
1 fatal error and 3 warnings found.
```
Comment 4 Dave Cottlehuber freebsd_committer 2015-03-02 23:00:32 UTC
It seems that the go dependencies that are downloaded are always latest commit on master branch. A not uncommon go packaging problem. I don't know what the right way to address this is, but clearly the current approach won't get us a repeatable build. It seems github.com/tools/godep has reasonable support. Is this going to be a suitable dependency to help manage this more effectively?

Sadly I'm away on 6 weeks vacation but may have time to look into this on return. Thanks for getting a working port in the first place!
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2015-03-02 23:57:36 UTC
Dependencies from master branches on Github is very very likely untenable for ports.

iirc, there was a fledgling implementation of a bsd.go.mk (possibly now in USES=go form) that aimed to address the question of Go dependency management in the ports framework.

CC'ing current lang/golang maintainer in on this, who may be able to provide more detail.
Comment 6 cheffo 2015-03-03 07:21:07 UTC
I'll fix the port to download the correct commits from github (I though I already did this, but looks like I misunderstood github "download" process). Thus creating a snapshot which should stay the same.

Of course it will be much easier if our ports framework does support better those repos (and multiple distfiles) better and a porter can only define commit tags and get stuff done.
Comment 7 cheffo 2015-03-03 07:59:35 UTC
This will fix the github download issue:

MASTER_SITES=   GITHUB:github   \
                https://codeload.github.com/BurntSushi/toml/legacy.tar.gz/443a628?dummy=/:toml \
                https://codeload.github.com/bmizerany/pat/legacy.tar.gz/b8a3500?dummy=/:pat \
                https://codeload.github.com/boltdb/bolt/legacy.tar.gz/a7d19d8?dummy=/:bolt \
                https://codeload.github.com/kimor79/gollectd/legacy.tar.gz/1d0fc88?dummy=/:gollectd \
                https://codeload.github.com/peterh/liner/legacy.tar.gz/d3fd034?dummy=/:liner \
                https://codeload.github.com/rakyll/statik/legacy.tar.gz/4a16c83?dummy=/:statik \


But no idea how to fix the issue with googlecode.com and googlesource.com
It's just drifting.

P.S. Dave - I really like this:
- I added a dedicated influxdb user account for cleaner security
- used /usr/sbin/daemon to manage influxd itself

Influx do not support background mode right now, so using /usr/sbin/daemon looks the right approach to me.
Comment 8 Dave Cottlehuber freebsd_committer 2015-03-03 08:16:07 UTC
AFAICT the issue is that `go get` $THING retrieves upstream packages afresh. I found the same issue when doing the OSX port for hugo, a go static site engine.

I'm not sure yet if /usr/ports/lang/go/files/bsd.go.mk correctly handles versioning of dependent components; I can't see how e.g. https://github.com/freebsd/freebsd-ports/blob/master/security/go.crypto/Makefile seems to get things right, as the bsd.go.mk file has no smarts.

Is the best option is to implement a port of godep, and then use a godep file as a patch that specifies the required dependencies?

@cheffo BTW I am on efnet/#freebsdports as dch if you want to discuss over IRC.

The best long-term solution is probably to extend bsd.go.mk so that it can simply be given a list of tuples for dependencies, extracted from Godep output, and it will Do The Right Thing wrt fetching and updating:

{
	"ImportPath": "github.com/influxdb/influxdb",
	"GoVersion": "go1.4.2",
	"Deps": [
		{
			"ImportPath": "github.com/boltdb/bolt",
			"Comment": "v1.0-43-gcf33c9e",
			"Rev": "cf33c9e0ca0a23509b8bb8edfc63e4776bb1a330"
		},
		{
			"ImportPath": "golang.org/x/crypto/bcrypt",
			"Rev": "1351f936d976c60a0a48d728281922cf63eafb8d"
		},
		{
			"ImportPath": "golang.org/x/crypto/blowfish",
			"Rev": "1351f936d976c60a0a48d728281922cf63eafb8d"
		}
	]
}

But that's beyond my skills at present.
Comment 9 cheffo 2015-03-03 08:39:55 UTC
I'm not sure about "go get". It updates only if you use -u (which I do not) or if the dependency is missing. Have in mind that poudriere build jails do not have network so if "go get" tries to fetch something the "tesport" will fail.

Something like godep as part of our go support will be great, but again I do not see how this will fix the "drift" issue with googlecode.
Comment 10 cheffo 2015-03-04 16:41:37 UTC
Created attachment 153788 [details]
influxdb090r7.shar

ChangeLog to previous shar archive:

- add influxd user/group
- Switch from DIST to PORT VERSION - see https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/makefile-naming.html
- change CATEGORY
- update bolt & liner to latest commit
- update influx to RC7
- fix Makefile and make it look more pretty - change hard-coded stuff with variables.
- switch from google-code to github to resolve size drift issue
- improve influxd rc.d script to use daemon - thanks to https://github.com/skunkwerks
Comment 11 cheffo 2015-03-04 16:42:13 UTC
Created attachment 153789 [details]
patch to add influxd user in ports tree
Comment 12 cheffo 2015-03-04 16:42:37 UTC
Created attachment 153790 [details]
patch to add influxd group in ports tree
Comment 13 cheffo 2015-03-04 16:43:19 UTC
Created attachment 153791 [details]
poudriere testport output
Comment 14 cheffo 2015-03-04 16:46:06 UTC
influxdb-0.9.0.r7.portlint.log:
WARN: Makefile: possible use of absolute pathname "/var/db/${PORTNAME}/".
WARN: Makefile: possible use of absolute pathname "/var/log/${PORTNAME}...".
WARN: Makefile: possible use of absolute pathname "/var/run/${PORTNAME}...".
FATAL: Makefile: category "local" must be listed first
1 fatal error and 3 warnings found.


You can ignore the FATAL error - it's because I keep my ports in "local".

I believe all known and discussed here issues are resolved with latest archive.
Comment 15 Dave Cottlehuber freebsd_committer 2015-03-04 22:00:00 UTC
+1 awesome work/teamwork :D

- builds ok for me on 8.4, 9.3, 10.1 all amd64
- ship it already please!

alternatively 0.9.0 official release might come soon, maybe worth waiting for that.
Comment 16 Andrey Cherkashin 2015-06-15 20:19:49 UTC
> fetch: http://distcache.FreeBSD.org/ports-distfiles/influxdb-0.9.0.r7.tar.gz: Not Found

Maybe make sense to use git tag after all? I.e. lock it to one of those RC builds.
Comment 17 cheffo 2015-06-16 08:11:08 UTC
http://influxdb.com/download/index.html - version 0.9 release is out.
Updating this port to 0.9 is (somewhere) in my TODO list :)
Comment 18 Andrey Cherkashin 2015-06-16 17:38:11 UTC
(In reply to cheffo from comment #17)
I can try making a patch for this port if it's way deep down in your TODO list...
Comment 19 cheffo 2015-06-23 15:03:56 UTC
Created attachment 158018 [details]
Influxdb v0.9.0 - release
Comment 20 cheffo 2015-06-23 15:04:30 UTC
Created attachment 158019 [details]
poudriere testport output
Comment 21 Steve Wills freebsd_committer 2015-06-23 19:12:51 UTC
Thanks for the work on this. Recently, support for multiple github repos was added to ports. It would be good if this port could use that. It should be easier to port and maintain that way. See sysutils/serf as one of many examples. I'll try to work on updating this to use that and post an updated patch, but if you get it done first, that's fine too.
Comment 22 cheffo 2015-06-24 08:52:28 UTC
Created attachment 158031 [details]
Updated version of the port which follows and uses new model for github.
Comment 23 cheffo 2015-06-24 08:53:15 UTC
Created attachment 158032 [details]
poudriere testport output attached.
Comment 24 cheffo 2015-06-24 08:55:19 UTC
influxdb-0.9.0.portlint.log:
WARN: Makefile: possible use of absolute pathname "/var/db/${PORTNAME}/".
WARN: Makefile: possible use of absolute pathname "/var/log/${PORTNAME}...".
WARN: Makefile: possible use of absolute pathname "/var/run/${PORTNAME}...".
0 fatal errors and 3 warnings found.


Btw to which mail list should I subscribe to follow those changes in ports three like this one for github? :)
Comment 25 Steve Wills freebsd_committer 2015-06-26 00:48:50 UTC
Created attachment 158061 [details]
Updated patch

Here's an updated patch that doesn't have mercurial or git in the build deps. They aren't necessary since we have ports grab all the deps and put them in place. Likewise, there's no need to 'go get' since we have the deps. We just need to build/install. I also made some other cleanups. If this looks OK to you, I'll commit it.
Comment 26 cheffo 2015-06-26 09:04:10 UTC
Yep, you are right we do not need mercurial and git as dependency now :)
I'm fine with the changes, feel free to commit.

Thanks!
Comment 27 commit-hook freebsd_committer 2015-06-26 12:52:06 UTC
A commit references this bug:

Author: swills
Date: Fri Jun 26 12:51:10 UTC 2015
New revision: 390632
URL: https://svnweb.freebsd.org/changeset/ports/390632

Log:
  databases/influxdb: create port

  An open-source, distributed, time series database with no external dependencies.

  WWW: http://influxdb.com

  PR:		198073
  Submitted by:	Stefan Lambrev <cheffo@freebsd-bg.org>

Changes:
  head/GIDs
  head/UIDs
  head/databases/Makefile
  head/databases/influxdb/
  head/databases/influxdb/Makefile
  head/databases/influxdb/distinfo
  head/databases/influxdb/files/
  head/databases/influxdb/files/influxd.in
  head/databases/influxdb/files/patch-cluster_client__pool.go
  head/databases/influxdb/files/patch-cluster_shard__writer.go
  head/databases/influxdb/pkg-descr
  head/databases/influxdb/pkg-plist