Bug 186653 - [NEW PORT] databases/monetdb: Multithreaded "column-oriented" SQL databasee
Summary: [NEW PORT] databases/monetdb: Multithreaded "column-oriented" SQL databasee
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: Normal Affects Only Me
Assignee: John Marino
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-02-11 14:40 UTC by jim
Modified: 2014-08-09 11:56 UTC (History)
1 user (show)

See Also:


Attachments
.shar (52.53 KB, text/plain)
2014-02-11 14:40 UTC, jim
no flags Details
Revised makefile (7.30 KB, text/plain)
2014-08-09 02:01 UTC, John Marino
no flags Details
Revised Makefile #2 (7.27 KB, text/plain)
2014-08-09 09:45 UTC, John Marino
no flags Details
Revised pkg-plist (35.27 KB, text/plain)
2014-08-09 09:46 UTC, John Marino
no flags Details
Revised pkg-plist #2 (34.95 KB, text/plain)
2014-08-09 11:06 UTC, John Marino
no flags Details
Revised Makefile #3 (7.47 KB, text/plain)
2014-08-09 11:08 UTC, John Marino
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description jim 2014-02-11 14:40:01 UTC
MonetDB is an open source column-oriented database management system
developed at the Centrum Wiskunde & Informatica (CWI) in the
Netherlands. It was designed to provide high performance on complex
queries against large databases, such as combining tables with hundreds
of columns and multi-million rows. MonetDB has been applied in
high-performance applications for data mining, online analytical
processing, geographic information systems, XML Query (XQuery), text
and multimedia retrieval.

WWW: http://www.monetdb.org/

portlint: OK

Redports: https://redports.org/buildarchive/20140211134101-53795

Requires a dedicated user. The following patch needs to be applied against /usr/ports/GIDS and
/usr/ports/UIDs before building:

Index: GIDs
===================================================================
--- GIDs	(revision 343653)
+++ GIDs	(working copy)
@@ -87,6 +87,7 @@
 cbsd:*:150:
 rbldns:*:153:
 trircd:*:154:
+monetdb:*:155:
 sfs:*:171:
 agk:*:172:
 polipo:*:173:
Index: UIDs
===================================================================
--- UIDs	(revision 343653)
+++ UIDs	(working copy)
@@ -94,6 +94,7 @@
 cbsd:*:150:150::0:0:Cbsd user:/nonexistent:/bin/sh
 rbldns:*:153:153::0:0:rbldnsd pseudo-user:/nonexistent:/usr/sbin/nologin
 trircd:*:154:154::0:0:& user:/usr/local/etc/tr-ircd:/usr/sbin/nologin
+monetdb:*:155:155::0:0:MonetDB user:/usr/local/monetdb5:/bin/sh
 sfs:*:171:171::0:0:Self-Certifying File System:/nonexistent:/usr/sbin/nologin
 agk:*:172:172::0:0:AquaGateKeeper:/nonexistent:/nonexistent
 polipo:*:173:173::0:0:polipo web cache:/nonexistent:/usr/sbin/nologin

Generated with FreeBSD Port Tools 0.99_11 (mode: new)
Comment 1 John Marino freebsd_committer freebsd_triage 2014-08-07 14:33:03 UTC
Hi, if you are still interested in having this port in FreeBSD, it may (or may not) need to be reworked to support stage, and it may need updating to other newer conventions such as "USES" which is expanding all time.
For staging, see http://lists.freebsd.org/pipermail/freebsd-ports-announce/2014-May/000080.html


Additionally, you need to provide some sort of quality assurance.    
In order of preference, we are looking for:

1) "poudriere testport" or "poudriere bulk -t" logs
2) Redports or tinderbox logs

Please provide an updated shar file and attach a test log.  Alternatively, please indicate if you are no longer interested in having this software in the Ports Collection and that we can close the PR.

Thanks!
Comment 2 jim 2014-08-07 15:46:34 UTC
(In reply to John Marino from comment #1)
> Hi, if you are still interested in having this port in FreeBSD, it may (or
> may not) need to be reworked to support stage, and it may need updating to
> other newer conventions such as "USES" which is expanding all time.
> For staging, see
> http://lists.freebsd.org/pipermail/freebsd-ports-announce/2014-May/000080.
> html
> 
> 
> Additionally, you need to provide some sort of quality assurance.    
> In order of preference, we are looking for:
> 
> 1) "poudriere testport" or "poudriere bulk -t" logs
> 2) Redports or tinderbox logs
> 
> Please provide an updated shar file and attach a test log.  Alternatively,
> please indicate if you are no longer interested in having this software in
> the Ports Collection and that we can close the PR.
> 
> Thanks!


It is staged, and a redports link is attached to the original submission.
Comment 3 John Marino freebsd_committer freebsd_triage 2014-08-07 15:59:02 UTC
I missed the redports link, moving to "patch-ready", thanks!
Comment 4 John Marino freebsd_committer freebsd_triage 2014-08-07 16:01:25 UTC
In any case, I'll note that after 5 months there's a good chance the shar is outdated.  I took a look and the other thing that jumped out is "EXTRACT_SUFX=	.tar.xz" which still works but should be "USES+= tar:xz" now.
Comment 5 John Marino freebsd_committer freebsd_triage 2014-08-09 01:59:10 UTC
Okay, I found numerous issues include bugs, not the least of which the tarballs didn't fetch (using /Latest is not a good idea), nor was INSTALL_TARGET set to install-strip, so nothing was stripped.  (stage-qa caught that)

I highly simplified the port, and I defined monetdb (UID 156, not 155 which was already taken).  However, it's not passing stage-qa.  This is precisely why redports simply isn't good enough.  I spent a LOT of time on this port.  I wish I could have committed it tonight.

I'm going to attach the latest Makefile in a minute.  For the pkg-plist, I changed %%SPHINX%% to %%SPHINXCLIENT%% to work with OPTIONS_SUB=yes in 4 places.


Can you decide what to do about the following errors?
Also, isn't having 8 (variable) categories is a bit much?  I haven't seen ports with more than 2 or 3 before.


====> Running Q/A tests (stage-qa)
Error: 'lib/python2.7/site-packages/monetdb/mapi.pyc' is referring to /wrkdirs/usr/ports/databases/monetdb/work/stage
Error: 'lib/python2.7/site-packages/monetdb/exceptions.pyc' is referring to /wrkdirs/usr/ports/databases/monetdb/work/stage
Error: 'lib/python2.7/site-packages/monetdb/__init__.pyc' is referring to /wrkdirs/usr/ports/databases/monetdb/work/stage
Error: 'lib/python2.7/site-packages/monetdb/control.pyc' is referring to /wrkdirs/usr/ports/databases/monetdb/work/stage
Error: 'lib/python2.7/site-packages/monetdb/sql/connections.pyc' is referring to /wrkdirs/usr/ports/databases/monetdb/work/stage
Error: 'lib/python2.7/site-packages/monetdb/sql/monetize.pyc' is referring to /wrkdirs/usr/ports/databases/monetdb/work/stage
Error: 'lib/python2.7/site-packages/monetdb/sql/cursors.pyc' is referring to /wrkdirs/usr/ports/databases/monetdb/work/stage
Error: 'lib/python2.7/site-packages/monetdb/sql/converters.pyc' is referring to /wrkdirs/usr/ports/databases/monetdb/work/stage
Error: 'lib/python2.7/site-packages/monetdb/sql/__init__.pyc' is referring to /wrkdirs/usr/ports/databases/monetdb/work/stage
Error: 'lib/python2.7/site-packages/monetdb/sql/pythonize.pyc' is referring to /wrkdirs/usr/ports/databases/monetdb/work/stage
Error: 'lib/python2.7/site-packages/monetdb/sql/types.pyc' is referring to /wrkdirs/usr/ports/databases/monetdb/work/stage
Warning: .la libraries found, port needs USES=libtool
*** Error code 1

Stop.
make: stopped in /usr/ports/databases/monetdb
Comment 6 John Marino freebsd_committer freebsd_triage 2014-08-09 02:01:45 UTC
Created attachment 145543 [details]
Revised makefile

Mostly simplified, but corrects a couple of bugs and changes the MASTER_SITE which was already obsolete.  It got rid of all the loops in install and post-install as they weren't necessary.
Comment 7 John Marino freebsd_committer freebsd_triage 2014-08-09 02:05:02 UTC
moving back to "open" status so Jim can fix the python errors.
Comment 8 John Marino freebsd_committer freebsd_triage 2014-08-09 09:45:25 UTC
Created attachment 145550 [details]
Revised Makefile #2

Adds missing USES+= libtool, condenses OPTIONS_* to a few lines
Comment 9 John Marino freebsd_committer freebsd_triage 2014-08-09 09:46:40 UTC
Created attachment 145551 [details]
Revised pkg-plist

Reflects %%SPHINXCLIENT%% and libtool changes
Comment 10 John Marino freebsd_committer freebsd_triage 2014-08-09 09:51:00 UTC
kwm tells me this is a known issue with a known workaround:
https://wiki.freebsd.org/ports/StageDir
title: Python scripts reference work or stage directory (in a complaint from the staging Q/A checks)
Comment 11 John Marino freebsd_committer freebsd_triage 2014-08-09 11:06:27 UTC
Created attachment 145552 [details]
Revised pkg-plist #2

change mainly fixes python options
Comment 12 John Marino freebsd_committer freebsd_triage 2014-08-09 11:08:59 UTC
Created attachment 145553 [details]
Revised Makefile #3

Python fixes; removes python 2.7 and 3.3 hardcoding
Comment 13 John Marino freebsd_committer freebsd_triage 2014-08-09 11:15:46 UTC
Besides the .pyc issue, there were other issues with python, namely python3 didn't work at all and it was hardcoded to version 3.3 both in the makefile and the pkg-plist.  This also required *a lot* of rework.

It passes all poudriere checks now on FreeBSD 10 / amd64.
Just to be complete, I'm going to re-run it through redports 8x.

This is a *very* complex port, and I think I just proved that deciding that poudriere wasn't required for verification wasn't a good idea.

The only reason I continued to spend time on this is because you had waited since February and were annoyed because you felt it was at a high quality and modern level.  I really should have thrown this back but I wanted to highlight the problems and illustrate why poudriere testing really is important on complex ports.
Comment 14 John Marino freebsd_committer freebsd_triage 2014-08-09 11:44:11 UTC
Redports is happy: https://redports.org/buildarchive/20140809113431-35967/
Comment 15 commit-hook freebsd_committer freebsd_triage 2014-08-09 11:51:00 UTC
A commit references this bug:

Author: marino
Date: Sat Aug  9 11:50:15 UTC 2014
New revision: 364428
URL: http://svnweb.freebsd.org/changeset/ports/364428

Log:
  Add new port databases/monetdb

  PR:		186653
  Submitted by:	Jim (Ohlste.in)
  Reworked by:	marino

  MonetDB is an open source column-oriented database management system
  developed at the Centrum Wiskunde & Informatica (CWI) in the
  Netherlands. It was designed to provide high performance on complex
  queries against large databases, such as combining tables with hundreds
  of columns and multi-million rows. MonetDB has been applied in
  high-performance applications for data mining, online analytical
  processing, geographic information systems, XML Query (XQuery), text
  and multimedia retrieval.

Changes:
  head/databases/Makefile
  head/databases/monetdb/
  head/databases/monetdb/Makefile
  head/databases/monetdb/distinfo
  head/databases/monetdb/files/
  head/databases/monetdb/files/monetdbd.in
  head/databases/monetdb/files/patch-common-utils-mutils-c
  head/databases/monetdb/files/patch-configure
  head/databases/monetdb/files/patch-tools-merovingian-client-monetdb-c
  head/databases/monetdb/pkg-descr
  head/databases/monetdb/pkg-plist
Comment 16 John Marino freebsd_committer freebsd_triage 2014-08-09 11:56:14 UTC
I reworked a lot as you can see.  I tried to be careful and I don't think I broke anything, but there are many options and I didn't runtime test anything.  Moreover, newer versions of monetdb have been released.

You may want to review my work and submit a new PR to bring monetdb up to the latest version.  Just don't use "/Latest" for the MASTER_SITES because that's not a static link and the fetch will eventually fail if it's used.

I hope everything works as expected and I also hope I wasn't too abrasive but I really did want to illustrate a number of weaknesses (and outright bugs) the port had.  

Thanks,
John