Bug 231048 - [new port] devel/hadoop3: Apache Map/Reduce Framework v3.2
Summary: [new port] devel/hadoop3: Apache Map/Reduce Framework v3.2
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Yuri Victorovich
Depends on:
Blocks: 237481
  Show dependency treegraph
Reported: 2018-08-31 05:17 UTC by Johannes Jost Meixner
Modified: 2019-06-15 03:34 UTC (History)
4 users (show)

See Also:

hadoop3 shar (50.56 KB, text/plain)
2018-08-31 05:17 UTC, Johannes Jost Meixner
no flags Details
Hadoop 3.2 shell archive (121.41 KB, text/plain)
2019-04-21 15:34 UTC, Johannes Jost Meixner
no flags Details
CRF on hadoop 3.2 (107.73 KB, text/plain)
2019-04-23 16:45 UTC, Johannes Jost Meixner
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Johannes Jost Meixner freebsd_committer 2018-08-31 05:17:56 UTC
Created attachment 196734 [details]
hadoop3 shar

SHAR attached adds a preliminary variant devel/hadoop3 to the portstree.

What isn't done yet:
* porting over setsid/ssid patches from hadoop2
* rc.d scripts need a lot of work
* etc/ samples need a lot of work
* testing actual map/reduce through YARN & friends
* lots of polishing, testing

What's done:
* testing that namenode + datanode can play nice together
* the package builds... to some degree

What would be a great idea before committing:

* renaming hadoop to hadoop1
* renaming hadoop3 to hadoop
* ensuring dependencies match

I am hoping to do a lot of these items within the next month(s) but would appreciate any help/patches/feedback given.
Comment 1 Johannes Jost Meixner freebsd_committer 2018-08-31 05:22:50 UTC
Two more things:

* the work/m2 directory needs to be hosted somewhere else, and DISTFILES :maven should  adjusted
* once that's done, the maven call in do-build should have `--offline` appended
Comment 2 Kurt Jaeger freebsd_committer 2018-08-31 17:46:04 UTC
I wasn't aware that it's possible to create a shar file that does not
create the directories required ?

x - hadoop3/files/zkfc.in
/tmp/had: cannot create hadoop3/files/zkfc.in: No such file or directory

Hmm. Any idea ?
Comment 3 Kurt Jaeger freebsd_committer 2018-08-31 18:08:39 UTC
Ok, I can cope with mkdir in small numbers 8-)

testbuild@work after I try to silence portlint by butching the makefile 8-}
Comment 4 Kurt Jaeger freebsd_committer 2018-09-01 08:00:07 UTC
testbuild failed on 11.1a, see

Comment 5 Kurt Jaeger freebsd_committer 2018-09-01 11:54:47 UTC
Fails with the same problem on cur:

Comment 6 Kurt Jaeger freebsd_committer 2018-09-03 19:04:07 UTC
Port is still being worked on, ETA end of October.
Comment 7 Johannes Jost Meixner freebsd_committer 2019-04-21 15:34:20 UTC
Created attachment 203866 [details]
Hadoop 3.2 shell archive

Bump to Hadoop 3.2 

- builds fine (for me, in poudriere testport)
- after adding configuration [1], namenode and datanode can be started
- I've carried over the sedsid->ssid patches from devel/hadoop2
- I've had to import and extend https://jira.apache.org/jira/browse/MAPREDUCE-6417

[1] https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html
Comment 8 Kurt Jaeger freebsd_committer 2019-04-23 05:35:40 UTC
portlint has some comments. One of them is that patch filenames longer
than 100 characters are FATAL. I have no idea on how to handle this.

Comment 9 Kurt Jaeger freebsd_committer 2019-04-23 15:46:27 UTC
Build fails on 11.2amd64, but as it's a new port, I guess that can be tolerated.


Build fails on current-i386, but I guess that's also expected.

I'll ask portmgr about the overlong filenames.
Comment 10 Tobias Kortkamp freebsd_committer 2019-04-23 15:59:08 UTC
(In reply to Johannes Jost Meixner from comment #7)
> - I've carried over the sedsid->ssid patches from devel/hadoop2

sysutils/ssid installs as setsid too since ports r482969, so this seems
unnecessary and some of the patches could be dropped.
Comment 11 Johannes Jost Meixner freebsd_committer 2019-04-23 16:45:10 UTC
Created attachment 203936 [details]
CRF on hadoop 3.2

- remove setsid/ssid patches
- fix @dir in pkg-plist / user:group info
- remove bogus .orig files from pkg-plist

Once I get a working arcanist setup on the buildbox I will close this PR and move everything to Phabricator. Might make reviewing / commenting things a lot easier - especially given that there are still a few renames and new ports forthcoming.

Thanks tobik@ and lwhsu@ for the feedback!
Comment 12 Johannes Jost Meixner freebsd_committer 2019-04-23 16:46:58 UTC
(In reply to Kurt Jaeger from comment #9)
I'm guessing the build failure is due to different libc versions.

I removed the USE_GCC line so it would build with Clang. Might have to add some
USES+= compiler
Comment 13 Kurt Jaeger freebsd_committer 2019-04-23 17:30:59 UTC
Comment 14 Kurt Jaeger freebsd_committer 2019-04-28 19:44:41 UTC
builds on current, test-build on 12 @work
Comment 15 Kurt Jaeger freebsd_committer 2019-04-28 19:52:56 UTC
Builds on 12.0, fails to build on 11.2:


Is it supposed to build on 11.2 or is that an expected failure ?
Comment 16 Johannes Jost Meixner freebsd_committer 2019-05-13 02:31:21 UTC
(In reply to Kurt Jaeger from comment #15)
So far I've really only tested it on 13.0-CURRENT, as that's what my buildbox runs.
Comment 17 Yuri Victorovich freebsd_committer 2019-06-12 16:03:53 UTC
On 12amd64 there's this plist error:
> ===> Checking for items in STAGEDIR missing from pkg-plist
> Error: Orphaned: @dir %%HADOOP_LOGDIR%%
> ===> Checking for items in pkg-plist which are not in STAGEDIR
>Error: Missing: @dir %%HADOOP_LOGDIR%%