Bug 231048 - [new port] devel/hadoop3: Apache Map/Reduce Framework v3.2
Summary: [new port] devel/hadoop3: Apache Map/Reduce Framework v3.2
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Yuri Victorovich
Depends on:
Blocks: 237481
  Show dependency treegraph
Reported: 2018-08-31 05:17 UTC by Johannes Jost Meixner
Modified: 2019-08-06 18:58 UTC (History)
4 users (show)

See Also:

hadoop3 shar (50.56 KB, text/plain)
2018-08-31 05:17 UTC, Johannes Jost Meixner
no flags Details
Hadoop 3.2 shell archive (121.41 KB, text/plain)
2019-04-21 15:34 UTC, Johannes Jost Meixner
no flags Details
CRF on hadoop 3.2 (107.73 KB, text/plain)
2019-04-23 16:45 UTC, Johannes Jost Meixner
no flags Details
CRF for hadoop 3.2 (130.53 KB, text/plain)
2019-07-27 20:20 UTC, Johannes Jost Meixner
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Johannes Jost Meixner freebsd_committer 2018-08-31 05:17:56 UTC
Created attachment 196734 [details]
hadoop3 shar

SHAR attached adds a preliminary variant devel/hadoop3 to the portstree.

What isn't done yet:
* porting over setsid/ssid patches from hadoop2
* rc.d scripts need a lot of work
* etc/ samples need a lot of work
* testing actual map/reduce through YARN & friends
* lots of polishing, testing

What's done:
* testing that namenode + datanode can play nice together
* the package builds... to some degree

What would be a great idea before committing:

* renaming hadoop to hadoop1
* renaming hadoop3 to hadoop
* ensuring dependencies match

I am hoping to do a lot of these items within the next month(s) but would appreciate any help/patches/feedback given.
Comment 1 Johannes Jost Meixner freebsd_committer 2018-08-31 05:22:50 UTC
Two more things:

* the work/m2 directory needs to be hosted somewhere else, and DISTFILES :maven should  adjusted
* once that's done, the maven call in do-build should have `--offline` appended
Comment 2 Kurt Jaeger freebsd_committer 2018-08-31 17:46:04 UTC
I wasn't aware that it's possible to create a shar file that does not
create the directories required ?

x - hadoop3/files/zkfc.in
/tmp/had: cannot create hadoop3/files/zkfc.in: No such file or directory

Hmm. Any idea ?
Comment 3 Kurt Jaeger freebsd_committer 2018-08-31 18:08:39 UTC
Ok, I can cope with mkdir in small numbers 8-)

testbuild@work after I try to silence portlint by butching the makefile 8-}
Comment 4 Kurt Jaeger freebsd_committer 2018-09-01 08:00:07 UTC
testbuild failed on 11.1a, see

Comment 5 Kurt Jaeger freebsd_committer 2018-09-01 11:54:47 UTC
Fails with the same problem on cur:

Comment 6 Kurt Jaeger freebsd_committer 2018-09-03 19:04:07 UTC
Port is still being worked on, ETA end of October.
Comment 7 Johannes Jost Meixner freebsd_committer 2019-04-21 15:34:20 UTC
Created attachment 203866 [details]
Hadoop 3.2 shell archive

Bump to Hadoop 3.2 

- builds fine (for me, in poudriere testport)
- after adding configuration [1], namenode and datanode can be started
- I've carried over the sedsid->ssid patches from devel/hadoop2
- I've had to import and extend https://jira.apache.org/jira/browse/MAPREDUCE-6417

[1] https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html
Comment 8 Kurt Jaeger freebsd_committer 2019-04-23 05:35:40 UTC
portlint has some comments. One of them is that patch filenames longer
than 100 characters are FATAL. I have no idea on how to handle this.

Comment 9 Kurt Jaeger freebsd_committer 2019-04-23 15:46:27 UTC
Build fails on 11.2amd64, but as it's a new port, I guess that can be tolerated.


Build fails on current-i386, but I guess that's also expected.

I'll ask portmgr about the overlong filenames.
Comment 10 Tobias Kortkamp freebsd_committer 2019-04-23 15:59:08 UTC
(In reply to Johannes Jost Meixner from comment #7)
> - I've carried over the sedsid->ssid patches from devel/hadoop2

sysutils/ssid installs as setsid too since ports r482969, so this seems
unnecessary and some of the patches could be dropped.
Comment 11 Johannes Jost Meixner freebsd_committer 2019-04-23 16:45:10 UTC
Created attachment 203936 [details]
CRF on hadoop 3.2

- remove setsid/ssid patches
- fix @dir in pkg-plist / user:group info
- remove bogus .orig files from pkg-plist

Once I get a working arcanist setup on the buildbox I will close this PR and move everything to Phabricator. Might make reviewing / commenting things a lot easier - especially given that there are still a few renames and new ports forthcoming.

Thanks tobik@ and lwhsu@ for the feedback!
Comment 12 Johannes Jost Meixner freebsd_committer 2019-04-23 16:46:58 UTC
(In reply to Kurt Jaeger from comment #9)
I'm guessing the build failure is due to different libc versions.

I removed the USE_GCC line so it would build with Clang. Might have to add some
USES+= compiler
Comment 13 Kurt Jaeger freebsd_committer 2019-04-23 17:30:59 UTC
Comment 14 Kurt Jaeger freebsd_committer 2019-04-28 19:44:41 UTC
builds on current, test-build on 12 @work
Comment 15 Kurt Jaeger freebsd_committer 2019-04-28 19:52:56 UTC
Builds on 12.0, fails to build on 11.2:


Is it supposed to build on 11.2 or is that an expected failure ?
Comment 16 Johannes Jost Meixner freebsd_committer 2019-05-13 02:31:21 UTC
(In reply to Kurt Jaeger from comment #15)
So far I've really only tested it on 13.0-CURRENT, as that's what my buildbox runs.
Comment 17 Yuri Victorovich freebsd_committer 2019-06-12 16:03:53 UTC
On 12amd64 there's this plist error:
> ===> Checking for items in STAGEDIR missing from pkg-plist
> Error: Orphaned: @dir %%HADOOP_LOGDIR%%
> ===> Checking for items in pkg-plist which are not in STAGEDIR
>Error: Missing: @dir %%HADOOP_LOGDIR%%
Comment 18 Johannes Jost Meixner freebsd_committer 2019-07-27 20:20:32 UTC
Created attachment 206104 [details]
CRF for hadoop 3.2

* improved DEPENDS: added LIB_DEPENDS on protobuf25
* IGNORE on FreeBSD 11: I can't figure out how to make C++'s getline definition (istream) play nice with C's getline definition (stdio.h) 
* Build tested w/ poudriere testport on 12-STABLE and 13-CURRENT
Comment 19 Yuri Victorovich freebsd_committer 2019-07-27 22:10:09 UTC
(In reply to Johannes Jost Meixner from comment #18)

It fails for me:
[WARNING] cd /usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfs && /usr/bin/cc -DLIBHDFS_DLL_EXPORT -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/javah -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target -I/usr/local/openjdk8/include -I/usr/local/openjdk8/include/freebsd -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/main/native -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/main/native/libhdfs -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/posix -O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -fno-strict-aliasing -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -fvisibility=hidden -o CMakeFiles/test_libhdfs_threaded_hdfs_static.dir/os/posix/thread.c.o   -c /usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/posix/thread.c
[WARNING] --- main/native/libhdfspp/lib/proto/CMakeFiles/protoc-gen-hrpc.dir/all ---
[WARNING] ld: error: undefined symbol: google::protobuf::compiler::CodeGenerator::GenerateAll(std::__1::vector<google::protobuf::FileDescriptor const*, std::__1::allocator<google::protobuf::FileDescriptor const*> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::compiler::GeneratorContext*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*) const
[WARNING] >>> referenced by protoc_gen_hrpc.cc
[WARNING] >>>               CMakeFiles/protoc-gen-hrpc.dir/protoc_gen_hrpc.cc.o:(vtable for StubGenerator)
[WARNING] c++: error: linker command failed with exit code 1 (use -v to see invocation)
[WARNING] *** [main/native/libhdfspp/lib/proto/protoc-gen-hrpc] Error code 1
[WARNING] make[4]: stopped in /usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target
[WARNING] 1 error
Comment 20 Johannes Jost Meixner freebsd_committer 2019-07-28 09:37:33 UTC
(In reply to Yuri Victorovich from comment #19)
Which os release is this, which linker are you using (ld.lld or ld.bfd) and is this in poudriere?
Comment 21 Yuri Victorovich freebsd_committer 2019-07-28 09:50:28 UTC
(In reply to Johannes Jost Meixner from comment #20)

FreeBSD 12.0-STABLE r347548

LLD 8.0.0 (FreeBSD 356365-1200007) (compatible with GNU linkers)

No poudriere.
Comment 22 Johannes Jost Meixner freebsd_committer 2019-07-28 09:58:14 UTC
I tested with 12-stable in poudriere, worked fine...

* Do you have the latest version of protobuf25 installed? In 2.5.0_5, antoine@ fixed it up yesterday and changed the LDCONFIG entry so it gets cached automatically. 

* I've seen this error countless times with rebuilds myself; did you have a clean tree (no work/ dir) before running `make` ?

You could try these two and build it again, I used `make clean package` many many times to get there.
Comment 23 Yuri Victorovich freebsd_committer 2019-07-28 10:33:36 UTC
(In reply to Johannes Jost Meixner from comment #22)

I just upgraded to protobuf25-2.5.0_5 and my build tree is clean, but it still fails.
Comment 24 Yuri Victorovich freebsd_committer 2019-07-28 10:51:36 UTC
It also failed in poudriere in check-plist: https://people.freebsd.org/~yuri/hadoop3-3.2.0.log
Comment 25 Johannes Jost Meixner freebsd_committer 2019-07-28 14:22:26 UTC
(In reply to Yuri Victorovich from comment #24)
Your log contains 581 "Orphaned:" entries.
My pkg-plist is 581 lines long. 

Are you sure you grabbed the latest copy? ;-)
Comment 26 Johannes Jost Meixner freebsd_committer 2019-08-06 18:58:01 UTC
(In reply to Yuri Victorovich from comment #24)
Were you able to test this with a fully-extracted SHAR including pkg-plist?