Current port version is 1.2.1 and marked as BROKEN. Any chance to unbreak it and update to 1.6 [1]? [1] https://github.com/tensorflow/tensorflow/releases
Hi, Would u mind to prepare a patch ?
Hi, I will try to get this port cranked up to 1.7.1.
There looks like a Bazel bug [0][1] that prevents building whatever version of TensorFlow. So far, I have tried py{27,34,35,36}-TensorFlow-{1.7.0,1.7.1,1.8.0} with Bazel-0.13.0 and all failed. I am going to test again with Bazel-0.12.0. [0]: https://github.com/tensorflow/tensorflow/issues/19015 [1]: https://github.com/bazelbuild/bazel/issues/5177
Tried Bazel-0.12.0 + TensorFlow-1.8.0. I am stuck on a problem regarding gRPC [0]; cannot proceed anymore. It looks like Bazel fetches the gRPC code on-the-fly, so there is no other way to perform the patch on our side (Makefile). Also, the combination of Bazel-0.12.0 and any version of TensorFlow less than 1.8.0 has another problem of detecting CPU architecture wrong [1][2] (trying to compile as 'armv7-a') so right now the only option left is waiting for gRPC fix. [0]: https://github.com/grpc/grpc/issues/15010 [1]: https://github.com/bazelbuild/bazel/issues/4794#issuecomment-376078059 [2]: https://github.com/tensorflow/tensorflow/issues/18564#issuecomment-385463643
(In reply to Hyun Hwang from comment #4) Hello Hyun, maybe the patches from the port devel/grpc might help you in that case. Here are my notes when I was working on that port for a little while: - patch grpc -> src/core/lib/grp/arena.cc (use patch from grpc port) - patch tensorflow/tools/build_info/gen_build_info.py (shebang fix) - patch tensorflow/tools/git/gen_git_source.py (shebang fix) - patch tensorflow/core/platform/posix/posix_file_system.cc (replace APPLE with FREEBSD) - patch line #350 in tensorflow/core/platform/env.cc (same as in arena.cc) My approach was at that time: - add the required files that are downloaded on the fly to the makefile (will be fun, because a lot of entries will be needed) - patch both ${WRKSRC}/WORKSPACE + ${WRKSRC}/tensorflow/workspace.bzl to give bazel the hint to search locally for the required files Maybe those notes will help you further to bring py-tensorflow again in a good shape. If you're interested, I can also attach my non-finished Makefile.
(In reply to Kai from comment #5) Hi Kai, Thank you for the lead! I will continue tomorrow. :)
Another linking nightmare encountered and I don't know where to look at to fix this; looks like Python is trying to find `backtrace(3)` function inside TensorFlow's shared object, not from libexecinfo. Error output to be uploaded soon. I've been continuously fixing link errors when they were encountered; the most frequent one was `-ldl not found`, which in Linux exists as a separate library while FreeBSD has it in libc. I have tracked all the occurrence of `-ldl` in linker options to clear this issue out. Next one I encountered is `unknown symbol backtrace_symbols_fd`, which in Linux does not require special linker flag while in FreeBSD does. I managed to fix this by adding link option `-lexecinfo` to ALL Python-related build rules.
Created attachment 193882 [details] Partial Poudriere log.
Created attachment 194004 [details] INCOMPLETE_science_py-tensorflow.shar Hi, I'm giving this upgrading process up, as I have no more time to pour in. I originally aimed to use TF on FreeBSD for my thesis, but there are too many obstacles for me to continue; I'm switching to Linux (alas!) to meet the deadline. Attached is the shell archive of what I've been doing. The Makefile is incomplete in the sense of do-install target because the do-build fails in the middle of the process. The pkg-plist file does not exist because I was building this port from scratch. I send my sincere apology to the original reporter. Hopefully, someone more skillful than me takes this over. Or maybe after I graduate and have more free time, then I might revisit this issue.
Maintainer has been reset.
Martin, I can take it if you don't mind. Yuri
Any updates on this PR? Should we just remove the port?
Tensorflow is too important to not have in the ports tree. It needs to be fixed somehow.
Note that Tensorflow 1.9 is out and 1.10 is around the corner. > Tensorflow is too important to not have in the ports tree. Well, viewed from the ports tree it is just a broken leaf port. It is this PR that still keeps it in.
(In reply to Rene Ladan from comment #14) I will make an attempt to fix it.
Pending upstream response for now: https://github.com/tensorflow/tensorflow/issues/21366
Ping?
It was removed on 2018-08-08. Hopefully, it will be re-added some day, when it will become buildable. Yuri
This needs to be a priority, as it is a pretty critical piece of software for many. Unfortunately, although I have built many custom versions (Ubuntu) the state of bazel on FreeBSD has left it impossible for me (with limited skills) to do a build. I will try to checkout 1.13.1 again, but hope is needed ))
(In reply to ed austin from comment #19) I have plans to look at this again. I have some leads that might help, just didn't have time to look again. Yuri
reopen
Have you got any further with this port. I have started work on creating a port, and while everything builds successfully, it is the final linking that fails. libtensorflow_framework.so.1: Undefined symbol "_ZN3Aws11Environment6GetEnvEPKc" I'm going to try to address the final linking error today: Here is my current work on the port, it's currently 1.14.0. One the build is successful, I have other things to address such as tidying up some patches and removing others in place of sed. Will also try and tackle killing the downloading and building of remote sources. https://github.com/Amzo/py-tensorflow
(In reply to Anthony from comment #22) Hi Anthony, I didn't have much time to work on tensorflow die to other things. If you'll be able to make it build I'll be more than happy to test and commit it. Thank you for working on the tensorflow port! Yuri
It currently builds fine now as I fixed the initial linking errors. However it fails on tensorflow lite while linking. I think this can safely be removed as tensorflow lite is for Android and IOS platforms. Some tinkering with the build files should ignore this being built. I'm guessing when it can't detect the current platform as MacOS, Linux or Windows it's defaulting to building this. So I will address this tomorrow.
(In reply to Anthony from comment #25) Sounds great! Please drop a line when you are done. Yuri
It now builds correctly for me using the Makefile and my patches. Still need to fix the installation and update the pkg-plist. I just need to push my last patch to git and then it should be good for testing.
It should be ready for testing. I had to rename the git repository I was using as I had to create additional packages as they weren't available in the ports but the latest tensorflow depends on them. absl and Gast both of which are available now and can be tested from here: https://github.com/Amzo/FreeBSD-Ports I'm not too familiar with creating ports so the Makefiles will need reviewed to ensure I've followed correct porting guidelines and naming. I will run a clean build in poudiere to ensure I haven't missed any dependencies which I may have already had installed on my system.
A commit references this bug: Author: yuri Date: Wed Jul 10 00:14:42 UTC 2019 New revision: 506324 URL: https://svnweb.freebsd.org/changeset/ports/506324 Log: New port: devel/py-absl: Abseil Python Common Libraries PR: 226400 Submitted by: Anthony <amzo1337@gmail.com> Changes: head/devel/Makefile head/devel/py-absl/ head/devel/py-absl/Makefile head/devel/py-absl/distinfo head/devel/py-absl/pkg-descr
A commit references this bug: Author: yuri Date: Wed Jul 10 00:29:33 UTC 2019 New revision: 506325 URL: https://svnweb.freebsd.org/changeset/ports/506325 Log: New port: devel/py-gast: AST that abstracts the underlying Python version PR: 226400 Submitted by: Anthony <amzo1337@gmail.com> Changes: head/devel/Makefile head/devel/py-gast/ head/devel/py-gast/Makefile head/devel/py-gast/distinfo head/devel/py-gast/pkg-descr
One problem that I noticed is that the build asks the interactive question: > Extracting Bazel installation... > WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown". > You have bazel 0.27.0 installed. > Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native -Wno-sign-compare]: This might be a bazel problem, not sure. The second problem is this: > Fetching @grpc; fetching 7s > Fetching @farmhash_archive; Restarting. The build should not fetch anything, this is not allowed. All needed files should appear in BUILD_DEPENDS and the build should use pre-downloaded versions. Bazel's BUILD files should be patched and all https:// URLs should be replaced with file:// URLS pointing to /usr/ports/distfiles or something like that. This is how bazel projects are difficult. Yuri
I already have plans to remove the downloading of the files via bazel, just wanted to make sure I had a buildable tensorflow first to avoid any problems and make troubleshooting more difficult. As for the optimization flags, I was going to add a port option to build native or for arch. Since tensorflow uses CPU optimizations this would be best for users building ports, but keep the ability to build for all amd64 machines so that it can be packaged still. As for bazel downloading files, I have a few solutions to this. The first is to create a tar file of what it downloads and have the ports Makefile download that from my git then modify the bazel files. This would be the easiest and quickest solution. Or could add the MASTERSITES to Makefiles but this would bloat too much.
(In reply to Anthony from comment #32) > As for bazel downloading files, I have a few solutions to this. The first is to create a tar file of what it downloads and have the ports Makefile download that from my git then modify the bazel files. This would be the easiest and quickest solution. No, the simpler solution is for these files to be added to MASTER_SITES/DISTFILES, and bazel's BUILD files to be patched accordingly.
(In reply to Yuri Victorovich from comment #33) Someone else attempted the port on Phabricator but people were unsure about going that route as the MASTER_SITES list would be huge for it. I can do this if this is how it should be done. If I kill the pulling and building of packages we already have in the ports such curl, jpeg, nasm and others. The MASTER_SITES could be reduced substantially and then I would only need to add google specific ones. I'll tackle this today and fix the pulling and downloading of the files.
(In reply to Anthony from comment #34) It doesn't matter that it will be huge. Anything that doesn't require a special file preparation in a special step is better.
I've updated the port on my github so now it uses system libraries and what external sources it needs to download are now downloaded via MASTERSITES. Since there was a few I created a MASTER_SITES file which the Makefile includes. Not sure if this is best practice or not. I also set the DIST_SUBDIR to ${PORTNAME}, as we need to copy these files to the WRKDIR for bazel and it needs the exact match to what is included in the workspace.bzl. Is there a way to rename these downloaded files? Also made it respect LOCALBASE, and patch BUILD files to find correct system binaries. E.G swig3.0. I needed to create a couple of new ports for the build to work. devel/double-conversion needed upgraded and I added: devel/crc32c, devel/nysnc, devel/google-cloud-cpp, devel/py-gast, devel/py-absl and devel/google-mock. Ran a test build in poudriere and all went well. Just need to add an option for cpu optimization and maybe create a tensorboard package. Since I'm still waiting to be accepted on reviews.freebsd.org. I'll just keep it posted here.
A commit references this bug: Author: yuri Date: Sun Jul 14 05:22:46 UTC 2019 New revision: 506603 URL: https://svnweb.freebsd.org/changeset/ports/506603 Log: New port: devel/nsync: C library that exports various synchronization primitives like mutexes PR: 226400 Submitted by: Anthony <amzo1337@gmail.com> Changes: head/devel/Makefile head/devel/nsync/ head/devel/nsync/Makefile head/devel/nsync/distinfo head/devel/nsync/pkg-descr head/devel/nsync/pkg-plist
A commit references this bug: Author: yuri Date: Sun Jul 14 05:32:17 UTC 2019 New revision: 506605 URL: https://svnweb.freebsd.org/changeset/ports/506605 Log: New port: devel/crc32c: CRC32C implementation supporting CPU-specific acceleration PR: 226400 Submitted by: Anthony <amzo1337@gmail.com> Changes: head/devel/Makefile head/devel/crc32c/ head/devel/crc32c/Makefile head/devel/crc32c/distinfo head/devel/crc32c/pkg-descr head/devel/crc32c/pkg-plist
I'm working on devel/google-cloud-cpp because the latest version fails to build. This may take a while.
What issues are there with google-cloud-cpp. The version 0.11.0 builds consistently at me, but I was going to spend some more time on tidying it up.
(In reply to Anthony from comment #40) No, please delay until I commit it.
Created attachment 205760 [details] google-cloud-cpp.shar The latest version of google-cloud-cpp installs files into the same directory where it builds from: https://github.com/googleapis/google-cloud-cpp/issues/2875 Attaching the latest revision of devel/google-cloud-cpp This is a show-stopper, and needs to be resolved in order to proceed. Waiting for the upstream to resolve this.
.build is where the files are downloaded to prevent the build system automatically downloading them. googleapis and nlohmann_json. Could move them to WRKSRC/.build. The original build automatically downloaded these source files. If I remember correctly that is where it placed those files.
(In reply to Anthony from comment #43) This issue has nothing to do with .build. They install files into the build directory. This needs to be fixed. Google probably works Mon-Fri, so we need to wait for them to fix this.
Oh right. Well I will continue to polish the tensorflow build as in poudriere in a clean build it isn't pulling in some dependencies. I probably messed up the depend lines as it's my first time making a port. So I will fix those.
Based on the information on the bug report I've patched the cmake files so the problem should be fixed. Instead of setting INSTALL_DIR to CMAKE_BIN_DIR where it is executed, install dir is now set to ${LOCALBASE} which fixes the issue until the build system is finished being reworked upstream. I also updated the pkg-descr. Here is the results of make makeplist after the patch. http://sprunge.us/2JJPDk
(In reply to Anthony from comment #46) Are you talking about https://github.com/Amzo/FreeBSD-Ports/blob/master/devel/google-cloud-cpp/Makefile ? It doesn't mention INSTALL_DIR or CMAKE_BIN_DIR.
Created attachment 205845 [details] Fix installing Files into bin dir Fixes installing the files into the build dir and installs them to the correct location.
(In reply to Anthony from comment #48) Hm, with the attached patch 'make makeplist' still shows wrong files: @dir /usr/ports/devel/google-cloud-cpp/work/.build/external @dir /usr/ports/devel/google-cloud-cpp/work/.build @dir /usr/ports/devel/google-cloud-cpp/work @dir /usr/ports/devel/google-cloud-cpp @dir /usr/ports/devel @dir /usr/ports Yuri
Is that with the modified Makefile from my github? I don't have the issue of installing wrong files anymore.
Sorry, I had forgot to push changes to my git, had only commited them.
Created attachment 205849 [details] google-cloud-cpp.shar I also made some changes against your port because CMAKE_OFF/CMAKE_ON wasn't used, and because the version wasn't right due to their mislabeling. This version has the issue in question. I am reluctant to throw in more time into making a workaround, when they admitted that the problem is on their end, and they are expected to fix it any time now. We shouldn't be "fixing" the port when there is nothing wrong with it. Yuri
I created the work around as they said it was marked as low priority and would be fixed when the build system got reworked which they had no ETA for and could be awhile. The fix is waiting on this issue. https://github.com/googleapis/google-cloud-cpp/issues/2802 I don't mind waiting, I just thought a work around would suffice until it is fixed upstream.
(In reply to Anthony from comment #53) Ok. So what exactly is the workaround here? The above patch alone doesn't fix it. What part of Makefile is also related to this workaround? Yuri
I added a post-patch: to the Makefile which goes with the patch I added: post-patch:  @${REINPLACE_CMD} "s#%%LOCALBASE%%#${LOCALBASE}#" \  ${WRKSRC}/cmake/external/googleapis.cmake I feel like i'm cluttering this bug report up.
A commit references this bug: Author: yuri Date: Wed Jul 17 20:36:59 UTC 2019 New revision: 506825 URL: https://svnweb.freebsd.org/changeset/ports/506825 Log: New port: devel/google-cloud-cpp: C++ Idiomatic Clients for Google Cloud Platform services PR: 226400 Submitted by: Anthony <amzo1337@gmail.com> Changes: head/devel/Makefile head/devel/google-cloud-cpp/ head/devel/google-cloud-cpp/Makefile head/devel/google-cloud-cpp/distinfo head/devel/google-cloud-cpp/files/ head/devel/google-cloud-cpp/files/patch-cmake_DownloadNlohmannJson.cmake head/devel/google-cloud-cpp/files/patch-cmake_GoogleCloudCppCommon.cmake head/devel/google-cloud-cpp/files/patch-cmake_external_googleapis.cmake head/devel/google-cloud-cpp/files/patch-google_cloud_storage_CMakeLists.txt head/devel/google-cloud-cpp/pkg-descr head/devel/google-cloud-cpp/pkg-plist
The pkg-descr file is wrong with that google-cloud-cpp submission. It's not the one from my github.com. https://github.com/Amzo/FreeBSD-Ports/blob/master/devel/google-cloud-cpp/pkg-descr "Google cloud C++ librariesC++ libraries for use with the Google Cloud platform. Google Cloud Platform provides infrastructure as a service, platform as a service, and serverless computing environments. WWW: https://github.com/googleapis/google-cloud-cpp"
(In reply to Anthony from comment #57) Sorry for this mishap, I will correct it.
A commit references this bug: Author: yuri Date: Wed Jul 17 22:33:57 UTC 2019 New revision: 506826 URL: https://svnweb.freebsd.org/changeset/ports/506826 Log: devel/google-cloud-cpp: Fix the wrong pkg-descr file that was committed PR: 226400 Reported by: Anthony <amzo1337@gmail.com> Changes: head/devel/google-cloud-cpp/pkg-descr
Tensorflow build fails for me: https://people.freebsd.org/~yuri/py36-tensorflow-1.14.0.log
Yeah, this is the issue i'm trying to fix. If I build tensorflow on my machine it builds fine. If I start a jail and build it it builds fine, but if I build it with poudriere it fails with that error. Those files are in /usr/local/include/google/ and I need to figure out what is stopping it from finding those files at times. Did you build it in a jail also or with poudriere. I'm trying to track down what is causing it, Once it does actually find the file sin ${LOCALBASE}/include the rest of the build works but I can only reproduce this sometimes and sometimes I can't. I've ruled out JOBS being the issues. As It works sometimes with --jobs said to 4 and sometimes it doesn't. Was that the latest port from github you tried as I made a few quite changes since I first posted it to dependencies and configurations as well as a couple of more patches.
(In reply to Anthony from comment #61) I tried the latest revision 9d91d3c from https://github.com/Amzo/FreeBSD-Tensorflow It failed for me both in the poudriere jail, and on my desktop system. Yuri
I'm goind to rework it because I did another test in poudriere and either TF_SYSTEM_LIBS isn't being passed, or I'm missing a dependency for the port that I have installed on my system but haven't included in the Makefile which would explain why it isn't building in poudriere for me but is on my host system. I'll look into it today. Might remove SETENV and add the variables for Configure into CONFIGURE_ARGS and set USE_CONFIGURE= yes. Once I find the culprit, fix and push it. I'll let you know.
I think I have found the culprit. When using TF_SYSTEM_LIBS it should symlink ${LOCALBASE}/include/google files to the build dir and include them but it is messing this up. The genrule to create these files is not being executed first but after. If you continue the build without cleaning it will continue. Even with jobs set to 1 it still doesn't create these files first. Need to find out why they aren't being generated first, before executing the protoc commands. It's possible there could be a check that checks for is_linux. so I'll dig into it and find a solution.
I beleive I have fixed it however this now requires jobs to be set to 1 and avoid parrallel processing. I wonder if the issue is because we're using normal make. I should maybe try setting USES = gmake and see if that also fixes it as I don't want to stop parallel building as it takes significantly longer to build. However it builds with out this now with a couple of more patches. I'll run the test build on host and in poudriere before pushing.
(In reply to Anthony from comment #65) Thanks for working on this! Yuri
(In reply to Anthony from comment #65) Anthony, If the only problem is that TensorFlow build fails on multiple CPUs, you can just disable multiple CPUs for now. Some ports do it this way. It is better to have it build on a single CPU, than be not available at all. Yuri
I've been building on a single core for now and seems to be working but it's slow for me as it's taking over 12 hours plus so could be awhile before I have things fully tested due to the slow compile time.
Thanks for your work on this, gents! It looks like you're nearing the finish. Any guess as to when the final commit might happen? We have multiple researchers with immediate interest in tensorflow, so let me know if there's anything I can do to help.
It may be a couple of days before I can look at it again as I'm currently working on something else also. Could you try a build with adding --jobs 1 to the bazel command in the Makefile you have to see if it builds for you also? As currently with --jobs 1 it builds for me outside of poudriere but still no luck within poudriere. It shouldn't be much longer as well as it does build and run as I've been testing it locally, it's just the issue of bazel running a command that needs to symlinks before creating the symlinks. I might try a build on Linux with our exact commands for building and exact dependencies and options and see if it is the same as it could be a potential bug. Bazel should really handle the dependency of it's building properly.
Are you still using anything from https://github.com/Amzo/FreeBSD-Tensorflow/tree/master/devel? I see that Yuri has done recent commits for everything except double-conversion. I attempted a build using committed versions of all dependencies and it didn't get far.
(In reply to Jason W. Bacon from comment #71) > I see that Yuri has done recent commits for everything except double-conversion. devel/double-conversion is already in the tree: https://www.freshports.org/devel/double-conversion/ Yuri
It needs the more recent version. Maybe I should add the check into the Makefile anyways so that it will inform if the correct version isn't installed. double-conversion>=3.5.1.
It should be submitted as a port update.
I just checked the HEAD branch for ports and double-conversion is still at version 3.0.0. Tensorflow needs version >=3.1.5.
(In reply to Anthony from comment #75) > double-conversion is still at version 3.0.0 bug#239367 waits for the maintainer's approval. You need to apply the patch attached to this bug until it is committed.
Using the double-conversion 3.1.5 from Anthony's repo, committed ports for everything else, and adding MAKE_JOBS_UNSAFE, it still fails early in the build for me. I would guess it's missing a dependency that's installed on your system if it builds for you. Are all of your packages and your ports tree up-to-date? ===> Building for py36-tensorflow-1.14.0_1 (cd /usr/ports/wip/py-tensorflow/work-py36/tensorflow-1.14.0 && bazel --bazelrc="/usr/ports/wip/py-tensorflow/work-py36/bazelrc" --output_user_root=/usr/ports/wip/py-tensorflow/work-py36/bazel_out build --host_copt="-I/usr/local/include" --host_linkopt="-L/usr/local/lib" --linkopt="-L/usr/local/lib" --config=opt --incompatible_no_support_tools_in_action_inputs=false --verbose_failures //tensorflow:libtensorflow.so //tensorflow:libtensorflow_cc.so //tensorflow:install_headers //tensorflow/tools/pip_package:build_pip_package) WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown". WARNING: /usr/ports/wip/py-tensorflow/work-py36/tensorflow-1.14.0/tensorflow/python/BUILD:3469:1: in py_library rule //tensorflow/python:standard_ops: target '//tensorflow/python:standard_ops' depends on deprecated target '//tensorflow/python/ops/distributions:distributions': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.distributions will not receive new features, and will be removed by early 2019. You should update all usage of `tf.distributions` to `tfp.distributions`. WARNING: /usr/ports/wip/py-tensorflow/work-py36/tensorflow-1.14.0/tensorflow/contrib/metrics/BUILD:16:1: in py_library rule //tensorflow/contrib/metrics:metrics_py: target '//tensorflow/contrib/metrics:metrics_py' depends on deprecated target '//tensorflow/python/ops/distributions:distributions': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.distributions will not receive new features, and will be removed by early 2019. You should update all usage of `tf.distributions` to `tfp.distributions`. WARNING: /usr/ports/wip/py-tensorflow/work-py36/tensorflow-1.14.0/tensorflow/python/BUILD:102:1: in py_library rule //tensorflow/python:no_contrib: target '//tensorflow/python:no_contrib' depends on deprecated target '//tensorflow/python/ops/distributions:distributions': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.distributions will not receive new features, and will be removed by early 2019. You should update all usage of `tf.distributions` to `tfp.distributions`. WARNING: /usr/ports/wip/py-tensorflow/work-py36/tensorflow-1.14.0/tensorflow/contrib/learn/BUILD:17:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': No longer supported. Switch to SavedModel immediately. WARNING: /usr/ports/wip/py-tensorflow/work-py36/tensorflow-1.14.0/tensorflow/contrib/learn/BUILD:17:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': No longer supported. Switch to SavedModel immediately. WARNING: /usr/ports/wip/py-tensorflow/work-py36/tensorflow-1.14.0/tensorflow/contrib/bayesflow/BUILD:17:1: in py_library rule //tensorflow/contrib/bayesflow:bayesflow_py: target '//tensorflow/contrib/bayesflow:bayesflow_py' depends on deprecated target '//tensorflow/contrib/distributions:distributions_py': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.contrib.distributions are unmaintained, unsupported, and will be removed by late 2018. You should update all usage of `tf.contrib.distributions` to `tfp.distributions`. WARNING: /usr/ports/wip/py-tensorflow/work-py36/tensorflow-1.14.0/tensorflow/contrib/BUILD:12:1: in py_library rule //tensorflow/contrib:contrib_py: target '//tensorflow/contrib:contrib_py' depends on deprecated target '//tensorflow/contrib/distributions:distributions_py': TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.contrib.distributions are unmaintained, unsupported, and will be removed by late 2018. You should update all usage of `tf.contrib.distributions` to `tfp.distributions`. INFO: Analyzed 4 targets (378 packages loaded, 17121 targets configured). INFO: Found 4 targets... INFO: From ProtoCompile tensorflow/core/debug/debugger_event_metadata.pb.cc: bazel-out/freebsd-opt/bin/external/protobuf_archive: warning: directory does not exist. ERROR: /usr/ports/wip/py-tensorflow/work-py36/tensorflow-1.14.0/tensorflow/core/BUILD:2255:1: ProtoCompile tensorflow/core/protobuf/autotuning.pb.cc failed (Exit 1) bazel-out/freebsd-opt/bin/external/protobuf_archive: warning: directory does not exist. google/protobuf/any.proto: File not found. google/protobuf/duration.proto: File not found. tensorflow/core/protobuf/autotuning.proto:10:1: Import "google/protobuf/any.proto" was not found or had errors. tensorflow/core/protobuf/autotuning.proto:11:1: Import "google/protobuf/duration.proto" was not found or had errors. tensorflow/core/protobuf/autotuning.proto:52:3: "google.protobuf.Duration" is not defined. tensorflow/core/protobuf/autotuning.proto:64:3: "google.protobuf.Any" is not defined. INFO: Elapsed time: 26.077s, Critical Path: 0.45s INFO: 3 processes: 3 local. FAILED: Build did NOT complete successfully *** Error code 1 Stop. make: stopped in /usr/ports/wip/py-tensorflow
Those files are in /usr/local/include/google/protobuf/. The genrule in the source at tensorflow-1.14.0/third_party/systemlibs/protobuf.BUILD is meant to create those files by creating sym links from the ones in ${LOCALBASE} but it isn't creating them before it is needed. I'll have to go through the dependencies one by one and try and rule out a missing dependency. If you resume the build again without cleaning, it will create the files and continue as normal.
Had a successful build on a 4-core Xeon system (4 jobs) after several restarts without cleaning. My slightly modified port (--jobs=5, use wip/double-conversion until upgrade is committed) is here: https://github.com/outpaddling/freebsd-ports-wip extracting: /usr/ports/wip/py-tensorflow/work-py36/tmp/tensorflow-1.14.0.dist-info/top_level.txt extracting: /usr/ports/wip/py-tensorflow/work-py36/tmp/tensorflow-1.14.0.dist-info/WHEEL extracting: /usr/ports/wip/py-tensorflow/work-py36/tmp/tensorflow-1.14.0.dist-info/METADATA extracting: /usr/ports/wip/py-tensorflow/work-py36/tmp/tensorflow-1.14.0.dist-info/RECORD cd /usr/ports/wip/py-tensorflow/work-py36/tmp && /bin/sh -c '(/usr/bin/find -Ed $1 $3 | /usr/bin/cpio -dumpl $2 >/dev/null 2>&1) && /usr/bin/find -Ed $1 $3 \( -type d -exec /bin/sh -c '\''cd '\''$2'\'' && chmod 755 "$@"'\'' . {} + -o -type f -exec /bin/sh -c '\''cd '\''$2'\'' && chmod 0644 "$@"'\'' . {} + \)' COPYTREE_SHARE tensorflow-1.14.0.dist-info /usr/ports/wip/py-tensorflow/work-py36/stage/usr/local/lib/python3.6/site-packages cd /usr/ports/wip/py-tensorflow/work-py36/tmp/tensorflow-1.14.0.data/purelib && /bin/sh -c '(/usr/bin/find -Ed $1 $3 | /usr/bin/cpio -dumpl $2 >/dev/null 2>&1) && /usr/bin/find -Ed $1 $3 \( -type d -exec /bin/sh -c '\''cd '\''$2'\'' && chmod 755 "$@"'\'' . {} + -o -type f -exec /bin/sh -c '\''cd '\''$2'\'' && chmod 0644 "$@"'\'' . {} + \)' COPYTREE_SHARE . /usr/ports/wip/py-tensorflow/work-py36/stage/usr/local/lib/python3.6/site-packages ====> Compressing man pages (compress-man) sed: /usr/ports/wip/py-tensorflow/work-py36/.PLIST.pymodtmp: No such file or directory ====> Running Q/A tests (stage-qa) Warning: you need USES=ssl Error: /usr/local/lib/python3.6/site-packages/tensorflow/compiler/tf2tensorrt/python/ops/libtftrt.so is linked to /usr/local/lib/libjpeg.so.8 from graphics/jpeg-turbo but it is not declared as a dependency Warning: you need USES+=jpeg
(In reply to Jason W. Bacon from comment #79) Should create a bug report for bazel about this, because this is bazel bugs that cause build failures.
I'm not sure where the issue is at. I can't tell if it's an issue with tensorflow or if it's an issue with bazel. It's possible it could be an issue with the FreeBSD bazel port. I'll run some more test before filing a bug report. I'll build it on linux with our exact configuration. At the minute running tensorflow build a with --spawn_strategy=standalone it causes each command to be executed as a sub process, but I don't think it is waiting for the processes to finish. I'm not really sure and I don't know where to start in debugging the build system. Tensorflow builds now but it's the system not building stuff in the correct order.
I need the latest revision on my git tested. The issue should be resolved and has worked fine in my test build in a fresh jail. I've also added an option for CPU_OPTIMISATION as by default it compiles with -march=native which prevent sit being portable so I've fixed this issue. Should I also spell it the American way in the Makefile? If you guys could test it now. The issue was the spawn strategy for bazel. By default it tries to spawn subcommands in a sandbox, and I had switched to standalone which is deprecated now. Switching the spawn strategy in files/bazelrc local where each subprocess is spawned locally seems to have fixed it.
(In reply to Anthony from comment #82) Ok, thank you, I am running it in poudriere. In your repository, you should also: * Rename MASTER_SITES -> Makefile.MASTER_SITES * Replace spaces with tabs in BUILD_DEPENDS lines * Change the first section to: > PORTNAME= tensorflow > DISTVERSIONPREFIX= v > DISTVERSION= 1.14.0 > CATEGORIES= science python > PKGNAMEPREFIX= ${PYTHON_PKGNAMEPREFIX} > EXTRACT_ONLY= tensorflow-tensorflow-${DISTVERSIONFULL}_GH0.tar.gz * Add > USE_GITHUB= yes
I will do, but getting failures again. In a clean build it worked fine after it build all the dependencies and succeeded with no intervention. Running make clean and rebuilding again results in the proto issues again. I have no idea what is going on here as it occasionally builds with -j1, -j4, -j8. Then it fails. I wonder if bazel is caching somewhere it's not mean to.
(In reply to Anthony from comment #84) Building: checking cached actions INFO: Found 4 targets... [0 / 857] [Prepa] Symlinking //tensorflow/tools/pip_package:build_pip_package INFO: From ProtoCompile tensorflow/core/lib/core/error_codes_pb2.py: bazel-out/freebsd-opt/bin/external/protobuf_archive/.: warning: directory does not exist. ERROR: /wrkdirs/usr/ports/science/py-tensorflow/work-py36/tensorflow-1.14.0/tensorflow/core/BUILD:2255:1: ProtoCompile tensorflow/core/protobuf/autotuning.pb.cc failed (Exit 1) google/protobuf/duration.proto: File not found. tensorflow/core/protobuf/autotuning.proto:11:1: Import "google/protobuf/duration.proto" was not found or had errors. tensorflow/core/protobuf/autotuning.proto:52:3: "google.protobuf.Duration" is not defined. INFO: Elapsed time: 15.234s, Critical Path: 0.14s INFO: 7 processes: 7 local. FAILED: Build did NOT complete successfully FAILED: Build did NOT complete successfully *** Error code 1
I'm filing a bug now for tensorflow to see if they can push me in the right direction.
Still fails for me: 1 warning generated. INFO: From ProtoCompile tensorflow/core/lib/core/error_codes.pb.cc: bazel-out/freebsd-opt/bin/external/protobuf_archive: warning: directory does not exist. ERROR: /usr/ports/wip/py-tensorflow/work-py36/tensorflow-1.14.0/tensorflow/core/protobuf/tpu/BUILD:25:1: ProtoCompile tensorflow/core/protobuf/tpu/optimization_parameters.pb.cc failed (Exit 1) bazel-out/freebsd-opt/bin/external/protobuf_archive: warning: directory does not exist. google/protobuf/wrappers.proto: File not found. tensorflow/core/protobuf/tpu/optimization_parameters.proto:5:1: Import "google/protobuf/wrappers.proto" was not found or had errors. tensorflow/core/protobuf/tpu/optimization_parameters.proto:8:3: "google.protobuf.FloatValue" is not defined. tensorflow/core/protobuf/tpu/optimization_parameters.proto:9:3: "google.protobuf.FloatValue" is not defined. INFO: Elapsed time: 214.784s, Critical Path: 50.73s INFO: 412 processes: 412 local. FAILED: Build did NOT complete successfully *** Error code 1 Stop. make: stopped in /usr/ports/wip/py-tensorflow
Current builds is going well now. I added pasta to TF_STSTEM_LIBS but didn't create a port for it. I've created a port for py-pasta and building tensorflow again. I'll run numerous builds though incase it's just a chance of it being lucky and succeeding.
Cool, I can test again as soon as your changes are pushed. Regarding optimi[sz]ation, I see 5 ports with British spelling and 193 with American. Up to you whether you want to be populist or correct. I'm American with some English descent, so I'm torn... ;-)
I have ran 5 new test builds and all worked fine. I ran 3 in host and 2 in poudriere. I haven't generated a pkg-plist yet which I'll do now. Instead of figuring out why it wasn't creating the symlinks to the header files in our build dir. I patched the build system to pass -I${LOCALBASE}/include. Ithis avoids the symlinks and uses the system headers anyways. I fixed the CPU optimisation flags and also tidied up the dependencies.
Forgot to mention. I created a new dependencies devel/py-google-pasta.
(In reply to Anthony from comment #90) Thank you for your work, Anthony! Yuri
A commit references this bug: Author: yuri Date: Mon Jul 29 07:03:11 UTC 2019 New revision: 507561 URL: https://svnweb.freebsd.org/changeset/ports/507561 Log: New port: devel/py-google-pasta: Library to refactor python code through AST manipulation PR: 226400 Submitted by: Anthony <amzo1337@gmail.com> Changes: head/devel/Makefile head/devel/py-google-pasta/ head/devel/py-google-pasta/Makefile head/devel/py-google-pasta/distinfo head/devel/py-google-pasta/pkg-descr
Yeah, it crossed my mind to suggest updating the include path rather than relying on symlinks, which seems rather hokey. Separate issue: Have you tried building on 11.2? I keep hitting this error, which I don't see on 12.0. Not sure yet if it's the OS version or something else. /usr/bin/clang -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/host/bin/tensorflow/core/_objs/protos_all_proto_cc_impl/device_attributes.pb.pic.d '-frandom-seed=bazel-out/host/bin/tensorflow/core/_objs/protos_all_proto_cc_impl/device_attributes.pb.pic.o' -fPIC -iquote . -iquote bazel-out/host/bin -iquote external/protobuf_archive -iquote bazel-out/host/bin/external/protobuf_archive -g0 -I/usr/local/include '-march=native' -g0 -Wno-unknown-warning-option -Wno-unused-but-set-variable -Wno-sign-compare -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c bazel-out/host/bin/tensorflow/core/framework/device_attributes.pb.cc -o bazel-out/host/bin/tensorflow/core/_objs/protos_all_proto_cc_impl/device_attributes.pb.pic.o) ERROR: /usr/ports/wip/py-tensorflow/work-py36/tensorflow-1.14.0/tensorflow/core/BUILD:2255:1: C++ compilation of rule '//tensorflow/core:autotuning_proto_cc_impl' failed (Exit 1) In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:230:34: error: expected member name or ';' after declaration specifiers ::PROTOBUF_NAMESPACE_ID::int32 major() const; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ /usr/include/sys/types.h:372:20: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:230:34: error: expected ')' /usr/include/sys/types.h:372:20: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:230:34: note: to match this '(' /usr/include/sys/types.h:372:19: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:230:34: error: expected ')' ::PROTOBUF_NAMESPACE_ID::int32 major() const; ^ /usr/include/sys/types.h:372:37: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:230:34: note: to match this '(' /usr/include/sys/types.h:372:25: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:230:34: error: C++ requires a type specifier for all declarations ::PROTOBUF_NAMESPACE_ID::int32 major() const; ^ /usr/include/sys/types.h:372:27: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:230:34: error: expected ')' /usr/include/sys/types.h:372:42: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:230:34: note: to match this '(' /usr/include/sys/types.h:372:24: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:236:34: error: expected member name or ';' after declaration specifiers ::PROTOBUF_NAMESPACE_ID::int32 minor() const; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ /usr/include/sys/types.h:373:20: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:236:34: error: expected ')' /usr/include/sys/types.h:373:20: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:236:34: note: to match this '(' /usr/include/sys/types.h:373:19: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:236:34: error: C++ requires a type specifier for all declarations ::PROTOBUF_NAMESPACE_ID::int32 minor() const; ^ /usr/include/sys/types.h:373:27: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:236:34: error: non-member function cannot have '&' qualifier /usr/include/sys/types.h:373:28: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:236:34: error: expected ')' /usr/include/sys/types.h:373:29: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:236:34: note: to match this '(' /usr/include/sys/types.h:373:24: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:370:34: error: expected member name or ';' after declaration specifiers ::PROTOBUF_NAMESPACE_ID::int32 major() const; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ /usr/include/sys/types.h:372:20: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:370:34: error: expected ')' /usr/include/sys/types.h:372:20: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:370:34: note: to match this '(' /usr/include/sys/types.h:372:19: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:370:34: error: expected ')' ::PROTOBUF_NAMESPACE_ID::int32 major() const; ^ /usr/include/sys/types.h:372:37: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:370:34: note: to match this '(' /usr/include/sys/types.h:372:25: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:370:34: error: C++ requires a type specifier for all declarations ::PROTOBUF_NAMESPACE_ID::int32 major() const; ^ /usr/include/sys/types.h:372:27: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:370:34: error: expected ')' /usr/include/sys/types.h:372:42: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:370:34: note: to match this '(' /usr/include/sys/types.h:372:24: note: expanded from macro 'major' #define major(x) ((int)(((u_int)(x) >> 8)&0xff)) /* major number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:376:34: error: expected member name or ';' after declaration specifiers ::PROTOBUF_NAMESPACE_ID::int32 minor() const; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ /usr/include/sys/types.h:373:20: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:376:34: error: expected ')' /usr/include/sys/types.h:373:20: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:376:34: note: to match this '(' /usr/include/sys/types.h:373:19: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:376:34: error: C++ requires a type specifier for all declarations ::PROTOBUF_NAMESPACE_ID::int32 minor() const; ^ /usr/include/sys/types.h:373:27: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ In file included from bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.cc:4: bazel-out/host/bin/tensorflow/core/protobuf/autotuning.pb.h:376:34: error: non-member function cannot have '&' qualifier /usr/include/sys/types.h:373:28: note: expanded from macro 'minor' #define minor(x) ((int)((x)&0xffff00ff)) /* minor number */ ^ fatal error: too many errors emitted, stopping now [-ferror-limit=] 20 errors generated. INFO: Elapsed time: 109.279s, Critical Path: 68.60s INFO: 638 processes: 638 local. FAILED: Build did NOT complete successfully *** Error code 1 Stop. make: stopped in /usr/ports/wip/py-tensorflow
I've only been running on 12.0. I'll set up a 11.2 jail and build. What clang version is in base on 11.2?
Build succeeded for me on 12.0. 11.2: FreeBSD unixdev1.hpc bacon ~ 1000: cc --version FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) Target: x86_64-unknown-freebsd11.2 Thread model: posix InstalledDir: /usr/bin Do you really need USE_PYTHON=distutils since you're manually performing configure, build and install? autoplist is probably not useful for sure.
I used the old tensorflow port and just continued the work on it. so they were already there. Will clear them all up after fixing the portlint issues. I also need to split the package as currently I build everything even the C api and python. This can be split to tensorflow and py-tensorflow but they need to share patches.
Created attachment 206152 [details] Test patch to rename major and minor to major_val and minor val on FreeBSD 11.x Could you try this patch for building on FreeBSD 11.x? the files in tensorflow/core/protobuf/autotuning.proto define major and minor but that is already used in sys/types.h Renaming the major and minor version for protobuf as major_val and minor_val. I'm running a build with the included patch on FreeBSD 11.2 at the minute and it's going well so far. Will rename the patch appropriately soon as it's just a test at the minute. I will set it to an extra-patch and have it apply only for versions of 11.x. After I test on 11.3 and 11.2.
I'm not seeing anything in the upstream build instructions to indicate that the C and Python APIs are meant to be built or installed separately: https://www.tensorflow.org/install/source Do you think splitting into two package with simplify things? At first glance, it looks to me like more complication.
Looking ahead to cleaning up the port: Is there a reason that the variables set with ${SETENV} under do-configure could not be assigned to CONFIGURE_ENV instead? That would be the more traditional approach. Just something to look at later after more pressing issues are resolved.
I tried with CONFIGURE_ENV but they didn't pass to bazel. Need setenv for Bazel. I was going to tidy the port up after fixing building. Get it to build on the versions 12.0 11.3 and 11.2 Tidy up the patches with make patch. Remove redundant parts of the Makefile from the original. Tidy up master sites. Fix packaging or split into two seperate ports. py-tensorflow and tensorflow as a possible slave port. The current Makefile builds everything Python and C++ API. Once those are sorted, then I can experiment with Intel's MKL-DNN as we have this in ports and OpenCL with SYSCL for some limited GPU support.
Only just saw your reply on the splitting of packages. It doesn't need to be split, but renaming the port to tensorflow instead of py-tensorflow would better represent this. ArchLinux splits the package into python-tensorflow and just tensorflow, but there isn't a need to do it. I figured it would be tidier.
Builds on FreeBSD 11.x now work for me in poudriere with the previous patch.
(In reply to Anthony from comment #103) We still have to wait until Sunday for bug#239367 timeout.
(In reply to Anthony from comment #101) When using CONFIGURE_ENV, did you set HAS_CONFIGURE? This works for me: https://github.com/outpaddling/freebsd-ports-wip/tree/master/py-tensorflow BTW, I'd be happy to make you a collaborator on the WIP (work-in-progress) collection: https://github.com/outpaddling/freebsd-ports-wip/blob/master/README We could then commit minor fixes to the same place and limit the chatter here to bigger issues. Regarding splitting the port: I would do whatever minimizes maintenance effort for the long-term. Man-hours are precious and there's plenty of other work to be done on ports.
Anthony, Please let me know if/when the tensorflow port is ready to be committed. IMO, there is value even if only a single-CPU build works, it can be committed like this too if this happens to be the case. Best, Yuri
Multiple job building is fixed and works now. The building on FreeBSD 11.2 11.3 and 12.0 also works. All that needs doing is tidying up the Makefile.
(In reply to Anthony from comment #107) Ok, I can tidy it up while committing. -) Yuri
Another thing is I have set it to for amd64 arch only, as I have no arm devices to test on and haven't had an i386 jail to test build but it may well compile on those if the CPU_OPTIMISATION is modified to accompany this.
Build worked for me with 32 threads, so I don't think we need to worry about this. Is there a plist now?
(In reply to Anthony from comment #109) Creating an i386 jail on amd64 is very easy: poudriere jail -c -j "$name" -v $release -a i386 I use port-poudriere-setup (part of port-dev) to semi-automatically create jails for {amd64,i386}-{12.x,11.x} all at once: https://github.com/outpaddling/freebsd-ports-wip/tree/master/port-dev
Still no plist yet, and I didn't try with an i386 jail as upstream says they only focus on 64bit support. So it may well work but will lack modern CPU optimisations and such.
I see. If upstream has officially abandoned 32-bit, I would definitely disable it with *_FOR_ARCHS.
You (In reply to Anthony from comment #112) BTW, if plist is too large, too unstable or too complex, it's better to use autoplist. Don't worry about it. If it builds now I will take it from here and just commit. Yuri
The build succeeded w/out poudriere. I will set it to autoplist (because plist is too large), and will build overnight in poudriere. Yuri
A commit references this bug: Author: yuri Date: Tue Aug 6 15:28:06 UTC 2019 New revision: 508257 URL: https://svnweb.freebsd.org/changeset/ports/508257 Log: Re-add science/py-tensorflow: Computation using data flow graphs for scalable machine learning Many thanks to Anthony who was highly persistent and skillful in getting the Tensorflow build on FreeBSD! PR: 226400 Submitted by: Anthony <amzo1337@gmail.com> Changes: head/MOVED head/science/Makefile head/science/py-tensorflow/ head/science/py-tensorflow/Makefile head/science/py-tensorflow/Makefile.MASTER_SITES head/science/py-tensorflow/distinfo head/science/py-tensorflow/files/bazelrc head/science/py-tensorflow/files/patch-WORKSPACE head/science/py-tensorflow/files/patch-bazelrc head/science/py-tensorflow/files/patch-tensorflow__third__party_grpc_include_grpc_impl_codegen_port__platform.h head/science/py-tensorflow/files/patch-tensorflow__third__party_protobuf_src_google_protobuf_compiler_plugin.pb.cc head/science/py-tensorflow/files/patch-tensorflow__third__party_protobuf_src_google_protobuf_compiler_plugin.pb.h head/science/py-tensorflow/files/patch-tensorflow_contrib_boosted__trees_build head/science/py-tensorflow/files/patch-tensorflow_contrib_ignite_kernels_client_ignite__plain__client__unix.cc head/science/py-tensorflow/files/patch-tensorflow_contrib_makefile_makefile head/science/py-tensorflow/files/patch-tensorflow_core_BUILD head/science/py-tensorflow/files/patch-tensorflow_core_platform_cloud_gcs__dns__cache.cc head/science/py-tensorflow/files/patch-tensorflow_core_platform_default_build__config.bzl head/science/py-tensorflow/files/patch-tensorflow_core_platform_posix_env.cc head/science/py-tensorflow/files/patch-tensorflow_core_proto_autotuning.proto head/science/py-tensorflow/files/patch-tensorflow_lite_experimental_micro_tools_make_targets_freebsd__makefile.inc head/science/py-tensorflow/files/patch-tensorflow_lite_kernels_internal_build head/science/py-tensorflow/files/patch-tensorflow_lite_tools_make_makefile head/science/py-tensorflow/files/patch-tensorflow_lite_tools_make_targets_freebsd__makefile.inc head/science/py-tensorflow/files/patch-tensorflow_tensorflow.bzl head/science/py-tensorflow/files/patch-tensorflow_tools_lib__package_build head/science/py-tensorflow/files/patch-tensorflow_tools_pip__package_build head/science/py-tensorflow/files/patch-tensorflow_workspace.bzl head/science/py-tensorflow/files/patch-third__party_aws_build.bazel head/science/py-tensorflow/files/patch-third__party_systemlibs_enum34.build head/science/py-tensorflow/files/patch-third__party_systemlibs_grpc.build head/science/py-tensorflow/files/patch-third__party_systemlibs_protobuf.bzl head/science/py-tensorflow/files/patch-third__party_systemlibs_swig.build head/science/py-tensorflow/files/patch-third__party_systemlibs_syslibs__configure.bzl head/science/py-tensorflow/files/patch-third_party_gpus_rocm_configure.bzl head/science/py-tensorflow/pkg-plist
Committed with some cosmetic changes. Thank you, Anthony, for your persistence in getting Tensorflow build!
If I need to make changes to the port such as adding new options, bug fixes etc. Where should I put those. I've been waiting over a month for approval on reviews.freebsd.org.
Thank you both for your work on a challenging port!
(In reply to Anthony from comment #118) In order to make a change you can create a bug report here, add me as CC, and I will be more than happy to commit them. (reviews.freebsd.org tends to have submissions stalled.) Yuri
Just need to clarify something here. When using tf.contrib with tensorflow it reports that it is missing tensorflow_estimator as it isn't bundle with 1.14.0 and should be packaged separately. However this would lead to a circular dependency as: tensorflow_estimator requires py-tensorflow as a BUILD_DEPENDENCY, but py-tensorflow requires tensorflow_estimator as a RUN_DEPENDENCY. Not sure how to go about fixing this. It doesn't affect core tensorflow usage only when using tensorflow contrib.
(In reply to Anthony from comment #121) Exactly *how* does tensorflow_estimator require tensorflow? Does tensorflow have to be installed? Or, does it just need access to the tensorflow source code? If it's the former, can tf.contrib be a separate port?
In Tensorflow 1.14.0 the estimator code was disabled for this release as the bundled version wasn't stable for 1.14.0. When making use of tf.contrib in python some reference tensorflow_estimator. The fix is simply making a package for tensorflow_estimator which I have already created. However, the build for tensorflow estimator requires tensorflow to build. And if you want to run the tf.contrib packages in tensorflow you also need tensorflow_estimator installed as a run dependency which would create a circular dependency in the ports. Gentoo handles this by having tensorflow_estimator as a Post merge dependency but ports have nothing like this. I mean it's not a big issue as it's mostly tensorflow v1 compat issues, which we have disabled by default in favour of the new v2. And in the next release of Tensorflow the contrib packages are planned to be removed. https://github.com/Amzo/FreeBSD-Tensorflow/tree/master/science/tensorflow-estimator
A commit references this bug: Author: yuri Date: Mon Aug 12 16:44:40 UTC 2019 New revision: 508752 URL: https://svnweb.freebsd.org/changeset/ports/508752 Log: New port: science/py-tensorflow-estimator: TensorFlow Estimator, an API to simplify machine learning programming Dependencies are unusual for science/py-tensorflow-estimator: science/py-tensorflow imports science/py-tensorflow-estimator in its python code, but science/py-tensorflow-estimator is a separate project that depends on science/py-tensorflow in order to build. PR: 226400 Submitted by: Anthony <amzo1337@gmail.com> Changes: head/science/Makefile head/science/py-tensorflow-estimator/ head/science/py-tensorflow-estimator/Makefile head/science/py-tensorflow-estimator/distinfo head/science/py-tensorflow-estimator/files/ head/science/py-tensorflow-estimator/files/patch-tensorflow__estimator_tools_pip__package_build__pip__package.sh head/science/py-tensorflow-estimator/pkg-descr head/science/py-tensorflow-estimator/pkg-plist