Bug 191633 - [new port] devel/storm - distributed realtime computation system used at twitter
Summary: [new port] devel/storm - distributed realtime computation system used at twitter
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Thomas Zander
URL: https://storm.incubator.apache.org/
Keywords:
Depends on:
Blocks:
 
Reported: 2014-07-05 16:05 UTC by Radim Kolar
Modified: 2014-08-31 19:57 UTC (History)
2 users (show)

See Also:


Attachments
port shar (3.51 KB, text/plain)
2014-07-05 16:05 UTC, Radim Kolar
no flags Details
Storm port shar (8.15 KB, text/plain)
2014-08-31 10:50 UTC, Radim Kolar
no flags Details
port shar file (8.15 KB, text/plain)
2014-08-31 15:59 UTC, Radim Kolar
no flags Details
port shar (16.59 KB, text/plain)
2014-08-31 19:28 UTC, Radim Kolar
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Radim Kolar 2014-07-05 16:05:11 UTC
Created attachment 144426 [details]
port shar

Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!

Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate.
Comment 1 John Marino freebsd_committer freebsd_triage 2014-07-31 16:13:54 UTC
This summary is much better!  I still need to request that you provide test logs from poudriere or redports or at least the output from "make check-plist" followed by "make stage-qa".
Comment 2 John Marino freebsd_committer freebsd_triage 2014-07-31 16:35:57 UTC
If you choose poudriere, make sure you use either "poudriere testport" or "poudriere bulk -t" in order to enable stage-qa checks, thanks!
Comment 3 Thomas Zander freebsd_committer freebsd_triage 2014-08-16 13:10:38 UTC
Could you ensure that portlint -CA does not show FATAL errors? Thanks!
Comment 4 John Marino freebsd_committer freebsd_triage 2014-08-24 19:45:50 UTC
PR is stuck.  portlint has fatal errors and no testlogs.
Comment 5 Radim Kolar 2014-08-30 16:22:15 UTC
my portlint do not show any errors

fbsd9:~/github-ports/storm> portlint -a
WARN: Makefile: [62]: possible use of "${CHMOD}" found. Use @owner/@group operators in pkg-plist instead.
0 fatal errors and 1 warning found.
Comment 6 John Marino freebsd_committer freebsd_triage 2014-08-30 16:39:46 UTC
I see this "		cd ${WRKSRC} && ${FIND} ${STORM_DIRS} ${STORM_CFG_DIRS} | \
X			${CPIO} --quiet -pdmu ${STAGEDIR}${DATADIR}"

That could probably be done better with COPYTREE_SHARE.  However, that's not a dealbreaker (I'd like the next committer decide)

However, testlogs please.

Thomas: can you try portlint again?  submitter says he can't reproduce
Comment 7 Thomas Zander freebsd_committer freebsd_triage 2014-08-30 18:30:50 UTC
Sure. It looks like there are files missing in the shar file. Look at the header:

# This archive contains:
#
#       storm/Makefile
#       storm/distinfo
#       storm/files
#       storm/pkg-descr

So it creates an empty files/ dir and the RC_SUBR referring files are missing:

devel/storm > portlint -CA
WARN: Makefile: [62]: possible use of "${CHMOD}" found. Use @owner/@group operators in pkg-plist instead.
FATAL: Makefile: storm-nimbus listed in USE_RC_SUBR, but files/storm-nimbus.in is missing.
FATAL: Makefile: storm-supervisor listed in USE_RC_SUBR, but files/storm-supervisor.in is missing.
FATAL: files: empty directory should be removed.
Comment 8 Radim Kolar 2014-08-31 10:50:23 UTC
Created attachment 146590 [details]
Storm port shar

reuploaded port shar file.
Comment 9 Thomas Zander freebsd_committer freebsd_triage 2014-08-31 14:42:44 UTC
Thanks for submitting the new shar. It builds and installs, but:
1) pkg notices that no architecture specific files are being installed. If this is correct (are the jar files identical on each platform?), please use NO_ARCH= yes
2) Symbolic link in do-install is most certainly wrong, points to a non-existing location
3) Do we really need dynamic plist generation? There are no OPTIONS and no files being installed that look specific to the environment. Can we please convert this to a static plist?
4) When using a static plist, the CHMOD should be removed and modes (if incorrect) should be set via @mode macro in pkg-plist
Comment 10 Radim Kolar 2014-08-31 15:59:14 UTC
Created attachment 146606 [details]
port shar file

added NO_ARCH
fixed symbolic link destination. it worked when port was submitted first

i prefer dynamic plist, its one less file to maintain
Comment 11 John Marino freebsd_committer freebsd_triage 2014-08-31 16:02:04 UTC
Thomas, FYI, dynamic plists are *heavily* frowned upon.

This needs very good justification.
Comment 12 Thomas Zander freebsd_committer freebsd_triage 2014-08-31 16:22:36 UTC
The advantage of static plists is that you as a maintainer can define what is supposed to be part of the package. Then if something goes wrong during the build/install process, this gets noticed. If you generate the plist with 'find *', the plist looks always correct even if it's not.

Please provide a static plist. There is no technical need for a dynamic one.
Thanks you!
Comment 13 Radim Kolar 2014-08-31 19:28:07 UTC
Created attachment 146614 [details]
port shar

use static pkg-plist
Comment 14 commit-hook freebsd_committer freebsd_triage 2014-08-31 19:56:58 UTC
A commit references this bug:

Author: riggs
Date: Sun Aug 31 19:55:57 UTC 2014
New revision: 366824
URL: http://svnweb.freebsd.org/changeset/ports/366824

Log:
  - Import storm - A distributed realtime computation system used at twitter
    version 0.9.2

  PR:		191633
  Submitted by:	hsn@sendmail.cz (maintainer)

Changes:
  head/devel/Makefile
  head/devel/storm/
  head/devel/storm/Makefile
  head/devel/storm/distinfo
  head/devel/storm/files/
  head/devel/storm/files/storm-nimbus.in
  head/devel/storm/files/storm-supervisor.in
  head/devel/storm/pkg-descr
  head/devel/storm/pkg-plist
Comment 15 Thomas Zander freebsd_committer freebsd_triage 2014-08-31 19:57:51 UTC
Committed, minus the makeplist target. With a pkg-plist file, we don't need it anymore :-)
Thanks!