Created attachment 196734 [details]
SHAR attached adds a preliminary variant devel/hadoop3 to the portstree.
What isn't done yet:
* porting over setsid/ssid patches from hadoop2
* rc.d scripts need a lot of work
* etc/ samples need a lot of work
* testing actual map/reduce through YARN & friends
* lots of polishing, testing
* testing that namenode + datanode can play nice together
* the package builds... to some degree
What would be a great idea before committing:
* renaming hadoop to hadoop1
* renaming hadoop3 to hadoop
* ensuring dependencies match
I am hoping to do a lot of these items within the next month(s) but would appreciate any help/patches/feedback given.
Two more things:
* the work/m2 directory needs to be hosted somewhere else, and DISTFILES :maven should adjusted
* once that's done, the maven call in do-build should have `--offline` appended
I wasn't aware that it's possible to create a shar file that does not
create the directories required ?
x - hadoop3/files/zkfc.in
/tmp/had: cannot create hadoop3/files/zkfc.in: No such file or directory
Hmm. Any idea ?
Ok, I can cope with mkdir in small numbers 8-)
testbuild@work after I try to silence portlint by butching the makefile 8-}
testbuild failed on 11.1a, see
Fails with the same problem on cur:
Port is still being worked on, ETA end of October.
Created attachment 203866 [details]
Hadoop 3.2 shell archive
Bump to Hadoop 3.2
- builds fine (for me, in poudriere testport)
- after adding configuration , namenode and datanode can be started
- I've carried over the sedsid->ssid patches from devel/hadoop2
- I've had to import and extend https://jira.apache.org/jira/browse/MAPREDUCE-6417
portlint has some comments. One of them is that patch filenames longer
than 100 characters are FATAL. I have no idea on how to handle this.
Build fails on 11.2amd64, but as it's a new port, I guess that can be tolerated.
Build fails on current-i386, but I guess that's also expected.
I'll ask portmgr about the overlong filenames.
(In reply to Johannes Jost Meixner from comment #7)
> - I've carried over the sedsid->ssid patches from devel/hadoop2
sysutils/ssid installs as setsid too since ports r482969, so this seems
unnecessary and some of the patches could be dropped.
Created attachment 203936 [details]
CRF on hadoop 3.2
- remove setsid/ssid patches
- fix @dir in pkg-plist / user:group info
- remove bogus .orig files from pkg-plist
Once I get a working arcanist setup on the buildbox I will close this PR and move everything to Phabricator. Might make reviewing / commenting things a lot easier - especially given that there are still a few renames and new ports forthcoming.
Thanks tobik@ and lwhsu@ for the feedback!
(In reply to Kurt Jaeger from comment #9)
I'm guessing the build failure is due to different libc versions.
I removed the USE_GCC line so it would build with Clang. Might have to add some
builds on current, test-build on 12 @work
Builds on 12.0, fails to build on 11.2:
Is it supposed to build on 11.2 or is that an expected failure ?
(In reply to Kurt Jaeger from comment #15)
So far I've really only tested it on 13.0-CURRENT, as that's what my buildbox runs.
On 12amd64 there's this plist error:
> ===> Checking for items in STAGEDIR missing from pkg-plist
> Error: Orphaned: @dir %%HADOOP_LOGDIR%%
> ===> Checking for items in pkg-plist which are not in STAGEDIR
>Error: Missing: @dir %%HADOOP_LOGDIR%%
Created attachment 206104 [details]
CRF for hadoop 3.2
* improved DEPENDS: added LIB_DEPENDS on protobuf25
* IGNORE on FreeBSD 11: I can't figure out how to make C++'s getline definition (istream) play nice with C's getline definition (stdio.h)
* Build tested w/ poudriere testport on 12-STABLE and 13-CURRENT
(In reply to Johannes Jost Meixner from comment #18)
It fails for me:
[WARNING] cd /usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfs && /usr/bin/cc -DLIBHDFS_DLL_EXPORT -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/javah -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target -I/usr/local/openjdk8/include -I/usr/local/openjdk8/include/freebsd -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/main/native -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/main/native/libhdfs -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/posix -O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -fno-strict-aliasing -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -fvisibility=hidden -o CMakeFiles/test_libhdfs_threaded_hdfs_static.dir/os/posix/thread.c.o -c /usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/posix/thread.c
[WARNING] --- main/native/libhdfspp/lib/proto/CMakeFiles/protoc-gen-hrpc.dir/all ---
[WARNING] ld: error: undefined symbol: google::protobuf::compiler::CodeGenerator::GenerateAll(std::__1::vector<google::protobuf::FileDescriptor const*, std::__1::allocator<google::protobuf::FileDescriptor const*> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::compiler::GeneratorContext*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*) const
[WARNING] >>> referenced by protoc_gen_hrpc.cc
[WARNING] >>> CMakeFiles/protoc-gen-hrpc.dir/protoc_gen_hrpc.cc.o:(vtable for StubGenerator)
[WARNING] c++: error: linker command failed with exit code 1 (use -v to see invocation)
[WARNING] *** [main/native/libhdfspp/lib/proto/protoc-gen-hrpc] Error code 1
[WARNING] make: stopped in /usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target
[WARNING] 1 error
(In reply to Yuri Victorovich from comment #19)
Which os release is this, which linker are you using (ld.lld or ld.bfd) and is this in poudriere?
(In reply to Johannes Jost Meixner from comment #20)
FreeBSD 12.0-STABLE r347548
LLD 8.0.0 (FreeBSD 356365-1200007) (compatible with GNU linkers)
I tested with 12-stable in poudriere, worked fine...
* Do you have the latest version of protobuf25 installed? In 2.5.0_5, antoine@ fixed it up yesterday and changed the LDCONFIG entry so it gets cached automatically.
* I've seen this error countless times with rebuilds myself; did you have a clean tree (no work/ dir) before running `make` ?
You could try these two and build it again, I used `make clean package` many many times to get there.
(In reply to Johannes Jost Meixner from comment #22)
I just upgraded to protobuf25-2.5.0_5 and my build tree is clean, but it still fails.
It also failed in poudriere in check-plist: https://people.freebsd.org/~yuri/hadoop3-3.2.0.log
(In reply to Yuri Victorovich from comment #24)
Your log contains 581 "Orphaned:" entries.
My pkg-plist is 581 lines long.
Are you sure you grabbed the latest copy? ;-)
(In reply to Yuri Victorovich from comment #24)
Were you able to test this with a fully-extracted SHAR including pkg-plist?