Bug 224584

Summary: [PATCH] lang/pocl: fix pkg-plist and update to POCL 1.3
Product: Ports & Packages Reporter: O. Hartmann <ohartmann>
Component: Individual Port(s)Assignee: Nikolai Lifanov <lifanov>
Status: Closed FIXED    
Severity: Affects Many People CC: brooks, jcfyecrayz, jmd, lantw44, lbartoletti, lifanov, marcel, ohartmann, rene, swills, w.schwarzenfeld, yuri
Priority: --- Keywords: patch
Version: Latest   
Hardware: Any   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233495
Bug Depends on:    
Bug Blocks: 231286    
Attachments:
Description Flags
POCL 1.0-patch and pkg-plist fix
none
Update of ohartman works
none
lang/pocl: unbreak and update to 1.0
none
Makefile adapting POCL 1.1
none
POCL-1.2-RC2
none
Update to POCL 1.2 Release
none
Update to POCL 1.2 official release
none
Update POCL 1.2 Release (with LLVM 7.0 as default backend)
none
pocl-1.3 patch (LLVM 8.0 as default)
none
pocl-1.3 patch (LLVM 8.0 as default) none

Description O. Hartmann 2017-12-25 15:07:00 UTC
Created attachment 189095 [details]
POCL 1.0-patch and pkg-plist fix

POCL  1.0 has been released. According to Bug 223032 there is a problem in pkg-plist, which is fixed with the workaround provided here: replace -unknown- by -portbld-, see patch diff.

The attached file is a svn diff against lang/pocl.

I've checked so far with poudriere to build the port lang/pocl and it built clear in my testenvironment.

I haven't looked deeper into the changes made by the developers (i.e. the CUDA backend).

Further testing is prohibited by Bug 223879 , Bug 219562 .
Comment 1 Bugzilla Automation freebsd_committer freebsd_triage 2017-12-25 15:07:00 UTC
Maintainer informed via mail
Comment 2 Yuri Victorovich freebsd_committer freebsd_triage 2017-12-25 17:52:11 UTC
*** Bug 224458 has been marked as a duplicate of this bug. ***
Comment 3 Loïc Bartoletti freebsd_committer freebsd_triage 2018-01-09 05:27:33 UTC
Hi,

I tried to build it without success (patch files aren't applied). Can you double check it? I will put you my patch based on your works.

But, there is another problem, pocl crash and tests failed ; maybe my patch is wrong.

Thanks.
Comment 4 Loïc Bartoletti freebsd_committer freebsd_triage 2018-01-09 05:29:06 UTC
Created attachment 189556 [details]
Update of ohartman works
Comment 5 Walter Schwarzenfeld freebsd_triage 2018-02-21 18:58:53 UTC
See also Bug #226092.
Comment 6 Nikolai Lifanov freebsd_committer freebsd_triage 2018-02-21 20:06:59 UTC
Created attachment 190869 [details]
lang/pocl: unbreak and update to 1.0

Hi! Have another patch from a duplicate issue (BZ#224584)

o unbreak lang/pocl at package time
o unbreak lang/pocl at runtime
o update lang/pocl to 1.0
o switch to fetching from Github
o switch to LLVM 5.0
o pet portlint

This was tested with security/hashcat on 12.0-CURRENT
Comment 7 Nikolai Lifanov freebsd_committer freebsd_triage 2018-02-21 20:07:34 UTC
*** Bug 226092 has been marked as a duplicate of this bug. ***
Comment 8 Nikolai Lifanov freebsd_committer freebsd_triage 2018-02-21 20:23:07 UTC
I'm going to take this bug.

O. Hartmann: what's still TODO to move this forward?
Comment 9 Johannes M Dieterich freebsd_committer freebsd_triage 2018-02-23 03:53:57 UTC
It definitely compiles and packages fine - so that's progress.

It also on executing clinfo look better (none of the previously reported issues) but hangs for me then. I am seeing this issue also elsewhere, so not sure that it's due to this.

I honestly think this can go in (assuming maintainer approval) as it is either way an improvement on the current situation. We can then sort whatever bugs appear subsequently?
Comment 10 O. Hartmann 2018-02-23 18:27:29 UTC
(In reply to Johannes M Dieterich from comment #9)

I have no objection with that, although the port isn't very healthy in this state - there is still a serious issue, please see Bug 219562 .

When lang/pocl is installed (with devel/ocl-icd) and graphics/blender has been compiled with CYCLES, blender (at the moment R2.79) fails with the same bug as reported in Bug 219562 . Ichecked. It is still present. Also, a very simple peice of code of my own, similar to clinfo, reporting the devices available in the system, fails with the error reported in the forementioned bug. See Bug 219562 for further informations.

Anyway, please do not rely on me in this matter! I think I'd go with Johannes M Dieterich opinion that it would be better to have a compiling port so far.

Regards
Oliver
Comment 11 O. Hartmann 2018-04-02 11:03:22 UTC
*** Bug 223032 has been marked as a duplicate of this bug. ***
Comment 12 O. Hartmann 2018-04-02 12:08:42 UTC
POCL 1.1 is out. But according to this bug:

http://lists.llvm.org/pipermail/llvm-bugs/2016-October/051237.html

the pocl tests fail and so the installation of the port due to this error message:

The following tests FAILED:
          1 - pocl_version_check (Failed)

The version can not be tested, since it fails with message:

1/197 Test   #1: pocl_version_check ..................................................................***Failed  Required regular expression not found.Regex=[basic]  0.02 sec

From POCL USER mailing list, I received word from Michal Babej upon this error and following his instructions (by fixing a csh/sh issue first in the appropriate script of the test performed):

[...]
Executing

# (sh ./tools/scripts/devel-envs.sh && clinfo)

gives me 

Unable to find symbol pthread_mutexattr_setkind_np version (null). Aborting.
Abort
[...]


Well, this, again, and again, seems to be related to this LLVM bug:

http://lists.llvm.org/pipermail/llvm-bugs/2016-October/051237.html

which I also mentioned in Bug 223032.

In my case, it seems graphics/belnder which requires LLVM 4.0 due to openshading language support, but the issue seems tricky as it occurs on FreeBSD whenever another OpenCL platform is also  backed via LLVM. In the case for POCL 1.1, I use LLVM 6.0. But the error also shows up with LLVM 5.0.

I'm out of ideas, time, mental capabilities. I's like to put the small effords I made (simply adopting the changes to reel in pocl 1.1 instead of pocl 1.0) here to the public, so someone can take a look. Due to the issue mentioned above (LLVM seems not to check for beeing the backend for multiple OpenCL platforms when dynamically linked to those platforms), POCL isn't working properly on FreeBSD with other OpenCL platforms selectable via an ICD like devel/opencl-icd. This sucks :-(
Comment 13 O. Hartmann 2018-04-02 12:13:21 UTC
Created attachment 192103 [details]
Makefile adapting POCL 1.1

The attached Makefile is a-work-in-progress to adopt POCL 1.1. It is not a replacement for the patch to move to our port to POCL-1.1 since tests fail and therefore the essential pkg-plist updates and checksums aren't in shape.

For your considerations.
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2018-04-02 12:37:29 UTC
*** Bug 224368 has been marked as a duplicate of this bug. ***
Comment 15 O. Hartmann 2018-04-11 09:02:44 UTC
I had to fix devel/ocl-icd first: somehow linking against pthreads wasn't there, so I had to add a configure option to make it working. Additionally to that, I tried to update to ocl-icd 2.2.12. Hopefully, the patch gets commited soon.

Now things regarding lang/pocl are getting a bit confusing. Since there is a serious problem according to a supposedly still remaining bug in LLVM (see http://lists.llvm.org/pipermail/llvm-bugs/2016-October/051237.html), updating to POCL 1.1 is now due to a test suite performed by the pocl source framework blocks installing POCL 1.1 as we know it from POCL 1.0.
Comment 16 O. Hartmann 2018-04-11 09:03:25 UTC
See Bug 226514.
Comment 17 O. Hartmann 2018-09-07 14:17:28 UTC
Created attachment 196942 [details]
POCL-1.2-RC2

For your consideration: POCL-1.2-RC2 adaption. Still seeing the bug: 

: CommandLine Error: Option 'enable-value-profiling' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
Comment 18 Yuri Victorovich freebsd_committer freebsd_triage 2018-09-21 06:07:39 UTC
What is the status of this?
Comment 19 O. Hartmann 2018-10-01 08:40:50 UTC
(In reply to Yuri Victorovich from comment #18)

POCL 1.2 has been releas on September, 25th. I try to check the port now with LLVM 7.0 as I have hope the nasty bug with multi-registering the very same LLVM backend, which prevents using an ICD (as devel/ocl-icd) has been solved - but I doubt that, since there is no report of any kind of mitigation.

My time- and barin-resources in this matter are limited, I regret.

VECMATH is going to be deprecated by POCL soon. Also, POCL 1.2 supports HWLOC2, FreeBSD doesn't support HWLOC 2, port devel/hwloc is still HWLOC 1. Maybe this could also be changed soon.

Attached, you'll find my latest attempt to maintain the port, with LLVM60 as the default.
Comment 20 O. Hartmann 2018-10-01 08:41:57 UTC
Created attachment 197659 [details]
Update to POCL 1.2 Release

Update to POCL 1.2, still LLVM60 as default.
Comment 21 O. Hartmann 2018-10-01 08:45:11 UTC
Created attachment 197660 [details]
Update to POCL 1.2 official release

My apology for the inconvenience, I needed to obsolete the old files.

The new patch incorporates most of the fixes done by developers herein and the small update by myself.
Comment 22 Brooks Davis freebsd_committer freebsd_triage 2018-10-01 23:07:24 UTC
(In reply to O. Hartmann from comment #21)

I tried this attachment and patch-lib_CL_devices_cpuinfo.c doesn't apply for me.
Comment 23 O. Hartmann 2018-10-02 07:04:15 UTC
Created attachment 197701 [details]
Update POCL 1.2 Release (with LLVM 7.0 as default backend)

Attached, you'll find a corrected patchset. It is for auditing purposes. Enabling LLVM 7.0 as the default backend doesn't solve the error:

X server found. dri2 connection failed! 
Device open failed, aborting...
: CommandLine Error: Option 'limited-coverage-experimental' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
Nothing to output !

This check was made by devel/clinfo.
Comment 24 John Hein 2018-11-24 18:16:58 UTC
(In reply to O. Hartmann from comment #23)
I get a similar error if lang/beignet and lang/clover are installed:

Simple test case:

#include <CL/opencl.h>
#include <stdio.h>
int
main()
{
    unsigned int uu;
    printf("rc: %d\n", clGetPlatformIDs(0, 0, &uu));
    printf("uu: %u\n", uu);
    return 0;
}

% cc -Wall -I/usr/local/include -L/usr/local/lib -lOpenCL cl.c -o cl
% ./cl
X server found. dri2 connection failed!
Device open failed, aborting...
: CommandLine Error: Option 'enable-value-profiling' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
Nothing to output !

clGetPlatformIDs() apparently is calling exit when it detectes what it thinks is an error (a very impolite think for a library to be doing).


Is it expected that beignet and clover should not be installed together for use with OpenCL?


See also bug 232357
Comment 25 John Hein 2018-11-25 00:53:39 UTC
See also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852746
Comment 26 Brooks Davis freebsd_committer freebsd_triage 2019-01-09 23:15:31 UTC
Is there any reason not to upgrade to LLVM 7.0?

AFACT, clinfo issues are irrelevant as it's either missing a required dependency to prevent multiple links to libllvm or it's broken by design.
Comment 27 Brooks Davis freebsd_committer freebsd_triage 2019-04-10 19:55:47 UTC
POCL 1.3 is out http://lists.llvm.org/pipermail/llvm-dev/2019-April/131561.html and supports llvm 8.0.  Support for llvm <6.0 will be remove in future releases per the release notes.
Comment 28 O. Hartmann 2019-04-13 08:33:42 UTC
I know, please be patient, update will come shortly.
Comment 29 O. Hartmann 2019-05-01 11:05:14 UTC
Created attachment 204140 [details]
pocl-1.3 patch (LLVM 8.0 as default)

Attached you'll find a patchfile for lifting pocl to version 1.3 and using by default LLVM 8.0.

The patches' DOCS option doesn't work for now; I also disabled all tests since I had trouble activating and running them properly. Later, I'd like to provide a WIP patchfile with TESTS also being an option.
Comment 30 O. Hartmann 2019-05-01 11:10:37 UTC
(In reply to Brooks Davis from comment #26)

clinfo is a simple way on pocl-supported FreeBSD platforms to show the pronlem reported in several bugs referenced in this thread.

POCL 1.3 in addition with LLVM80 as the backend still emits the error:

: CommandLine Error: Option 'limited-coverage-experimental' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options


when ocl-icd is used and other device backends, like beignet or clover are installed.
Comment 31 O. Hartmann 2019-05-01 12:18:35 UTC
I just searched the net for the problem reported with more than one .icd linked to the same LLVM backend (as the aforementioned error indicates),here are some pointing threads I found:

LLVM bug:
https://bugs.llvm.org/show_bug.cgi?id=30587

POCL:
https://github.com/pocl/pocl/issues/474

and another tool (here: Linux):
https://github.com/fireice-uk/xmr-stak/issues/1782

The LLVM bug is open since 2016 - almost three years now. As far as I can remember, the problem has been introduced shortly after LLVM 3.

POCL once offered a CMAKE arg like -DLLVM_STATIC, replaced by:

-DSINGLE_LLVM_LIB when this option is enabled (default), pocl tries to link to a single big LLVM library (libLLVM-<VERSION>.suffix). If this fails, it fallbacks to linking LLVM libraries provided by llvm-config --libfiles.

(source: http://portablecl.org/docs/html/install.html)

Enriching the above patchfile with 

CMAKE_ARGS+= -DSINGLE_LLVM_LIB=OFF (it is ON by default as the documentation stats, see above) does not have any influence of the broken behaviour.
Comment 32 O. Hartmann 2019-06-05 12:21:55 UTC
Created attachment 204838 [details]
pocl-1.3 patch (LLVM 8.0 as default)

After some notifications reported that lang/beignet has been marked DEPRECATED, I started again looking after POCL.

It seems POCL 1.3 compiles fine with -std=c++11, so I got rid of _CXXSTD=gnu++11.

After the deletion of lang/beignet from all FreeBSD CURRENT systems I run I was finally able to run "clinfo" successfully without any errors.


Please check.
Comment 33 commit-hook freebsd_committer freebsd_triage 2019-06-06 21:07:55 UTC
A commit references this bug:

Author: brooks
Date: Thu Jun  6 21:07:43 UTC 2019
New revision: 503602
URL: https://svnweb.freebsd.org/changeset/ports/503602

Log:
  Update to POCL 1.3.

  PR:		224584
  Submitted by:	ohartmann@walstatt.org (maintainer)

Changes:
  head/lang/pocl/Makefile
  head/lang/pocl/distinfo
  head/lang/pocl/files/patch-CMakeLists.txt
  head/lang/pocl/files/patch-config.h.in.cmake
  head/lang/pocl/files/patch-lib_CL_devices_cpuinfo.c
  head/lang/pocl/files/patch-lib_CL_pocl__binary.c
  head/lang/pocl/files/patch-tests_regression_test__issue__445.cpp
  head/lang/pocl/pkg-descr
  head/lang/pocl/pkg-plist
Comment 34 O. Hartmann 2019-06-09 08:28:45 UTC
There is a problem with the recent lang/pocl pkg-plist description in the ports tree compared to the patch I sent.

My local/personal poudriere fails to register the port after building with this error:

[...]
=======================<phase: package        >============================
===>  Building package for pocl-1.3
pkg-static: Unable to access file /wrkdirs/usr/ports/lang/pocl/work/stage/usr/local/share/pocl/kernel-x86_64-portbld-freebsd12.2-avx.bc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/lang/pocl/work/stage/usr/local/share/pocl/kernel-x86_64-portbld-freebsd12.2-avx2.bc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/lang/pocl/work/stage/usr/local/share/pocl/kernel-x86_64-portbld-freebsd12.2-avx512.bc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/lang/pocl/work/stage/usr/local/share/pocl/kernel-x86_64-portbld-freebsd12.2-avx_f16c.bc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/lang/pocl/work/stage/usr/local/share/pocl/kernel-x86_64-portbld-freebsd12.2-avx_fma4.bc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/lang/pocl/work/stage/usr/local/share/pocl/kernel-x86_64-portbld-freebsd12.2-sse2.bc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/lang/pocl/work/stage/usr/local/share/pocl/kernel-x86_64-portbld-freebsd12.2-sse41.bc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/lang/pocl/work/stage/usr/local/share/pocl/kernel-x86_64-portbld-freebsd12.2-ssse3.bc:No such file or directory
*** Error code 1
[...]

On my CURRENT as well as my 12-STABLE poudriere platform, there is no line like 

"%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.2-avx.bc" 

in pkg-plist as it is now in the ports tree. When running "make makeplist", it is always

"%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.0-avx.bc" 

(no .2, it is .0 as it is in the patchfile I've sent.). Did I miss something?
Comment 35 Yuri Victorovich freebsd_committer freebsd_triage 2019-06-10 01:30:23 UTC
(In reply to O. Hartmann from comment #34)

I'm getting this error too.
Comment 36 O. Hartmann 2019-06-11 15:17:21 UTC
(In reply to Yuri Victorovich from comment #35)

The commited patch deviates from the patchfile I sent in that way, that in pkg-plist, the rows

[...]
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.0-avx.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.0-avx2.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.0-avx512.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.0-avx_f16c.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.0-avx_fma4.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.0-sse2.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.0-sse41.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.0-ssse3.bc

has been (magically?) replaced by 

[...]
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.2-avx.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.2-avx2.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.2-avx512.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.2-avx_f16c.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.2-avx_fma4.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.2-sse2.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.2-sse41.bc
%%DATADIR%%/kernel-%%ARCH%%-portbld-%%PYTHON_PLATFORM%%.2-ssse3.bc

On my platforms running make/poudriere, %%PYTHON_PLATFORM%% expands to either freebsd11, freebsd12 or freebsd13 and the port appends always ".0"; I did not digg deeper into this, but I guess it's always ".0" on FreeBSD (while it might be relevant on Linux or other platforms I do not care for).

Be aware of the fact that the documentation is not build as it it suggested by the port. I have some trouble solving the problem due to time constraints.
Comment 37 Brooks Davis freebsd_committer freebsd_triage 2019-06-11 22:33:46 UTC
(In reply to O. Hartmann from comment #36)

It was .2 for me.  I don't know why.  I'll look into this tomorrow when my test machine is hopefully back online (building power outage).
Comment 38 commit-hook freebsd_committer freebsd_triage 2019-06-12 19:58:17 UTC
A commit references this bug:

Author: brooks
Date: Wed Jun 12 19:57:21 UTC 2019
New revision: 504035
URL: https://svnweb.freebsd.org/changeset/ports/504035

Log:
  Correct plist file.

  The previous entries stem from trusting over-zealous substitution in
  check-plist output.

  PR:		224584

Changes:
  head/lang/pocl/Makefile
  head/lang/pocl/pkg-plist
Comment 39 Rene Ladan freebsd_committer freebsd_triage 2019-06-30 10:44:55 UTC
Closing this for PR 231286 (llvm40 removal), I believe this port is fixed? Feel free to open a new PR if not ;)