Created attachment 196734 [details] hadoop3 shar SHAR attached adds a preliminary variant devel/hadoop3 to the portstree. What isn't done yet: * porting over setsid/ssid patches from hadoop2 * rc.d scripts need a lot of work * etc/ samples need a lot of work * testing actual map/reduce through YARN & friends * lots of polishing, testing What's done: * testing that namenode + datanode can play nice together * the package builds... to some degree What would be a great idea before committing: * renaming hadoop to hadoop1 * renaming hadoop3 to hadoop * ensuring dependencies match I am hoping to do a lot of these items within the next month(s) but would appreciate any help/patches/feedback given.
Two more things: * the work/m2 directory needs to be hosted somewhere else, and DISTFILES :maven should adjusted * once that's done, the maven call in do-build should have `--offline` appended
I wasn't aware that it's possible to create a shar file that does not create the directories required ? [...] x - hadoop3/files/zkfc.in /tmp/had: cannot create hadoop3/files/zkfc.in: No such file or directory Hmm. Any idea ?
Ok, I can cope with mkdir in small numbers 8-) testbuild@work after I try to silence portlint by butching the makefile 8-}
testbuild failed on 11.1a, see http://people.freebsd.org/~pi/logs/devel__hadoop3-111.txt
Fails with the same problem on cur: http://people.freebsd.org/~pi/logs/devel__hadoop3-cur.txt
Port is still being worked on, ETA end of October.
Created attachment 203866 [details] Hadoop 3.2 shell archive Bump to Hadoop 3.2 - builds fine (for me, in poudriere testport) - after adding configuration [1], namenode and datanode can be started - I've carried over the sedsid->ssid patches from devel/hadoop2 - I've had to import and extend https://jira.apache.org/jira/browse/MAPREDUCE-6417 [1] https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html
portlint has some comments. One of them is that patch filenames longer than 100 characters are FATAL. I have no idea on how to handle this. testbuilds@work.
Build fails on 11.2amd64, but as it's a new port, I guess that can be tolerated. https://people.freebsd.org/~pi/logs/devel__hadoop3-112-1555997211.txt Build fails on current-i386, but I guess that's also expected. I'll ask portmgr about the overlong filenames.
(In reply to Johannes Jost Meixner from comment #7) > - I've carried over the sedsid->ssid patches from devel/hadoop2 sysutils/ssid installs as setsid too since ports r482969, so this seems unnecessary and some of the patches could be dropped.
Created attachment 203936 [details] CRF on hadoop 3.2 - remove setsid/ssid patches - fix @dir in pkg-plist / user:group info - remove bogus .orig files from pkg-plist Once I get a working arcanist setup on the buildbox I will close this PR and move everything to Phabricator. Might make reviewing / commenting things a lot easier - especially given that there are still a few renames and new ports forthcoming. Thanks tobik@ and lwhsu@ for the feedback!
(In reply to Kurt Jaeger from comment #9) I'm guessing the build failure is due to different libc versions. I removed the USE_GCC line so it would build with Clang. Might have to add some USES+= compiler magic!
testbuilds@work
builds on current, test-build on 12 @work
Builds on 12.0, fails to build on 11.2: https://people.freebsd.org/~pi/logs/devel__hadoop3-112.txt Is it supposed to build on 11.2 or is that an expected failure ?
(In reply to Kurt Jaeger from comment #15) So far I've really only tested it on 13.0-CURRENT, as that's what my buildbox runs.
On 12amd64 there's this plist error: > ===> Checking for items in STAGEDIR missing from pkg-plist > Error: Orphaned: @dir %%HADOOP_LOGDIR%% > ===> Checking for items in pkg-plist which are not in STAGEDIR >Error: Missing: @dir %%HADOOP_LOGDIR%%
Created attachment 206104 [details] CRF for hadoop 3.2 * improved DEPENDS: added LIB_DEPENDS on protobuf25 * IGNORE on FreeBSD 11: I can't figure out how to make C++'s getline definition (istream) play nice with C's getline definition (stdio.h) * Build tested w/ poudriere testport on 12-STABLE and 13-CURRENT
(In reply to Johannes Jost Meixner from comment #18) It fails for me: [WARNING] cd /usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfs && /usr/bin/cc -DLIBHDFS_DLL_EXPORT -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target/native/javah -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target -I/usr/local/openjdk8/include -I/usr/local/openjdk8/include/freebsd -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/main/native -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/main/native/libhdfs -I/usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/posix -O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -fno-strict-aliasing -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -fvisibility=hidden -o CMakeFiles/test_libhdfs_threaded_hdfs_static.dir/os/posix/thread.c.o -c /usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/posix/thread.c [WARNING] --- main/native/libhdfspp/lib/proto/CMakeFiles/protoc-gen-hrpc.dir/all --- [WARNING] ld: error: undefined symbol: google::protobuf::compiler::CodeGenerator::GenerateAll(std::__1::vector<google::protobuf::FileDescriptor const*, std::__1::allocator<google::protobuf::FileDescriptor const*> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::compiler::GeneratorContext*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*) const [WARNING] >>> referenced by protoc_gen_hrpc.cc [WARNING] >>> CMakeFiles/protoc-gen-hrpc.dir/protoc_gen_hrpc.cc.o:(vtable for StubGenerator) [WARNING] c++: error: linker command failed with exit code 1 (use -v to see invocation) [WARNING] *** [main/native/libhdfspp/lib/proto/protoc-gen-hrpc] Error code 1 [WARNING] [WARNING] make[4]: stopped in /usr/ports/devel/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/target [WARNING] 1 error [WARNING]
(In reply to Yuri Victorovich from comment #19) Which os release is this, which linker are you using (ld.lld or ld.bfd) and is this in poudriere?
(In reply to Johannes Jost Meixner from comment #20) FreeBSD 12.0-STABLE r347548 LLD 8.0.0 (FreeBSD 356365-1200007) (compatible with GNU linkers) No poudriere.
I tested with 12-stable in poudriere, worked fine... Ideas: * Do you have the latest version of protobuf25 installed? In 2.5.0_5, antoine@ fixed it up yesterday and changed the LDCONFIG entry so it gets cached automatically. * I've seen this error countless times with rebuilds myself; did you have a clean tree (no work/ dir) before running `make` ? You could try these two and build it again, I used `make clean package` many many times to get there.
(In reply to Johannes Jost Meixner from comment #22) I just upgraded to protobuf25-2.5.0_5 and my build tree is clean, but it still fails.
It also failed in poudriere in check-plist: https://people.freebsd.org/~yuri/hadoop3-3.2.0.log
(In reply to Yuri Victorovich from comment #24) Your log contains 581 "Orphaned:" entries. My pkg-plist is 581 lines long. Are you sure you grabbed the latest copy? ;-)
(In reply to Yuri Victorovich from comment #24) Were you able to test this with a fully-extracted SHAR including pkg-plist? Thanks, Johannes
Same issue here with protobuf: [WARNING] --- main/native/libhdfspp/lib/proto/CMakeFiles/protoc-gen-hrpc.dir/all --- [WARNING] /usr/local/bin/ld: CMakeFiles/protoc-gen-hrpc.dir/protoc_gen_hrpc.cc.o: in function `StubGenerator::EmitMethod(google::protobuf::MethodDescriptor const*, google::protobuf::io::Printer*) const': [WARNING] /root/hej/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/proto/protoc_gen_hrpc.cc:83: undefined reference to `google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' [WARNING] /usr/local/bin/ld: CMakeFiles/protoc-gen-hrpc.dir/protoc_gen_hrpc.cc.o: in function `StubGenerator::EmitService(google::protobuf::ServiceDescriptor const*, google::protobuf::io::Printer*) const': [WARNING] /root/hej/hadoop3/work/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/proto/protoc_gen_hrpc.cc:64: undefined reference to `google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' [WARNING] collect2: error: ld returned 1 exit status protobuf version installed via pkg: Name : protobuf25 Version : 2.5.0_5
The version (3.4-SNAPSHOT) in trunk is targeting an upgrade of protobuf: https://issues.apache.org/jira/browse/HADOOP-13363 that might help with all the protobuf issues.
(In reply to Mårten Lindblad from comment #28) It worked with a new jail so I think the errors come from both having protobuf and protobuf25 installed.