Bug 247666 - games/anki: Update to new build system
Summary: games/anki: Update to new build system
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Kai Knoblich
Depends on:
Reported: 2020-06-30 19:00 UTC by David Schlachter
Modified: 2020-10-26 14:21 UTC (History)
2 users (show)

See Also:
kai: maintainer-feedback+

maturin.patch (42.97 KB, patch)
2020-07-05 09:28 UTC, Kai Knoblich
no flags Details | Diff
anki-2.1.33-preview.patch (192.93 KB, patch)
2020-09-09 23:25 UTC, Kai Knoblich
no flags Details | Diff
anki-2.1.33-73-preview.patch (191.78 KB, patch)
2020-09-11 18:53 UTC, Kai Knoblich
no flags Details | Diff
anki-2.1.35.patch (144.74 KB, patch)
2020-10-26 13:52 UTC, Kai Knoblich
kai: maintainer-approval+
Details | Diff
py-maturin.patch (47.77 KB, patch)
2020-10-26 13:53 UTC, Kai Knoblich
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description David Schlachter 2020-06-30 19:00:56 UTC
As noted in the ports commit history, Anki uses a new set of build scripts since version 2.1.17 (Jan 2020). I opened this bug report to organize notes about updating the port to the new build system.

Changes necessary to build:

- set USE_GITHUB to get latest source release
- use gmake instead of make
- bash required to build
- change all hardcoded paths to bash in the build scripts from /bin/bash to /usr/local/bin/bash (about 15 occurrences)
- set the PYTHON_BIN make environment variable to the actual python binary's name (rather than default "python3" in Makefile) (can be set with ports variable PYTHON_CMD?)

Changes that might be needed:

- build script attempts to fetch a pyqt5 5.13.2 with pip, which seemed to fail. Maybe add as a dependency and install with ports instead?
- build script attempts to install maturin with pip, which fails when building native module with rust ('building wheel'). However, continuous integration for maturin shows successful FreeBSD builds:
I tried building maturin according to the steps in cirrus.yml and it built fine. Maybe we need a maturin port?

Other possible changes that might be needed (need to investigate):

- prefix isn't specified in build script anymore, not sure how this is handled.
Comment 1 Kai Knoblich freebsd_committer 2020-07-05 09:21:26 UTC
(In reply to David Schlachter from comment #0)

Thank you for the notes. I add my findings/notes that have accumulated over the past few months.  Here's a rough overview of the build process (Anki 2.1.26): 

The build process consists of four main steps where the order is important:

1. Build the Anki Rust library in ${WRKSRC}/rslib 
        - Bootstrap/Setup Rust via rustup because it's bound to a nightly release (= 2020-02-27)
        - Create a ".so" file that is used by step 2

2. Build Rust/Python bridge in ${WRKSRC}/rspy
        - Bootstrap/Setup Rust via rustup because it's bound to a nightly release (= 2020-02-27)
        - Use maturin (yet to be ported) to create a Python compatible ".so" file (packaged in a Python Wheel)
        - For Python 3.7 the ".so" would be ankirspy.cpython-37m.so

3. Build/Install the Python library code in ${WRKSRC}/pylib to ${PYTHONPREFIX_SITELIBDIR}
        - The setup.py has a reference to "ankirspy" (created in step 2):

> install_requires.append("ankirspy==2.1.26")

        - The package is also available via PyPI -> anki 

4. Build/Install the Python Qt code in ${WRKSRC}/qt to ${PYTHONPREFIX_SITELIBDIR}
        - The package is also available via PyPI -> aqt

Before the build begins, some preparations are made. Most of them opens a network connection hence they must be done before the whole build process, e.g. via the "post-extract" and "patch" targets:

- Echo the actual build hash to ${WRKSRC}/meta/buildhash
        - Buildhash needs to be determined beforehand via "git rev-parse --short=8 HEAD"
        - Alternatively the string "dev" can be used as a build hash

- Fetch latest translation files (via Git) and place them to
        - ${WRKSRC}/rslib/ftl/repo
        - ${WRKSRC}/qt/ftl/repo
        - ${WRKSRC}/qt/po/repo

- Bootstrap Rust via "rustup" (downloads and setups a pre-compiled Rust environment)

- Determine/Sync/Download the Cargo crates
        - Both ${WRKSRC}/rslib and ${WRKSRC}/rspy directories have no Cargo.lock files
        - Cargo.toml needs to be patched -> remove vendored OpenSSL: 

> [target.'cfg(linux)'.dependencies]
> reqwest = { version = "0.10.1", features = ["json", "socks", "stream", "native-tls-vendored"] }

- Install the PyQt5 dependencies (if required for build).

The biggest challenge is at the moment to get a successful compilation of the Rust code as it works only with a specific rust-nightly release that is bootstrapped/setup via "rustup".

Here's a short overview:

>                                 | Anki 2.1.26           | HEAD branch (Commit 04b1ca75)
> --------------------------------+-----------------------+------------------------------
> rust 1.44.1_1                   | fails                 | Partially OK [1]
> --------------------------------+-----------------------+------------------------------
> rust-nightly    | fails                 | Partially OK [2]
> --------------------------------+-----------------------+------------------------------
> rust-nightly    | Partially OK [2]      | not tested
> --------------------------------+-----------------------+------------------------------
> rustup                          | OK                    | OK
> --------------------------------+-----------------------+------------------------------

[1] rslib builds fine but rspy fails because it uses a "pyo3" crate which can only be compiled with rust-nightly at the moment
[2] rslib builds fine but rspy fails because the rust-nightly date cannot be determined (maybe this can be patched out).
Comment 2 Kai Knoblich freebsd_committer 2020-07-05 09:28:52 UTC
Created attachment 216211 [details]

(In reply to David Schlachter from comment #0)

Attached is a draft of devel/maturin that should build fine and can be used to build the Rust/Python library.
Comment 3 PauAmma freebsd_triage 2020-08-14 22:59:02 UTC
Anki has lately taken to nagging me about 2.1.30 on startup.
Comment 4 Macos Edgar 2020-08-16 03:40:23 UTC
Comment 5 Kai Knoblich freebsd_committer 2020-09-09 23:25:43 UTC
Created attachment 217860 [details]

Here's a patch for preview and testing which updates Anki to 2.1.33. It also contains a slightly changed version of devel/py-maturin, which is also required to build the port.

Current open items:

- Although 2.1.33 is the latest version at the moment at this writing, an info will appear on startup that there's a new version available.

If you all have some spare cycles to build and test Anki 2.1.33 that would be great.
Comment 6 Kai Knoblich freebsd_committer 2020-09-11 18:53:58 UTC
Created attachment 217891 [details]

Attached is a patch for Anki 2.1.33-73 which uses a more recent commit (of the upcoming 2.1.34 release) which fixes some issues with synchronizing via AnkiWeb and migrating existing configuration files from older installations:

> Traceback (most recent call last):
>   File "/usr/local/share/anki/aqt/sync.py", line 93, in on_timer
>     on_normal_sync_timer(mw)
>   File "/usr/local/share/anki/aqt/sync.py", line 74, in on_normal_sync_timer
>     progress = mw.col.latest_progress()
>   File "/usr/local/share/anki/anki/collection.py", line 97, in
> latest_progress
>     return Progress.from_proto(self.backend.latest_progress())
>   File "/usr/local/share/anki/anki/rsbackend_gen.py", line 28, in
> latest_progress
>     output.ParseFromString(self._run_command(1, input))
>   File "/usr/local/share/anki/anki/rsbackend.py", line 262, in _run_command
>     err_bytes = bytes(e.args[0])
> IndexError: tuple index out of range

Thanks to David for the detailed error reports and tests!
Comment 7 David Schlachter 2020-10-05 21:44:51 UTC
Version 2.1.35 came out recently. To test, I manually generated anki-2.1.35-node_modules.tgz, updated the distinfo, and bumped the version in the Makefile:

# Patch to Makefile
< DISTVERSION=	2.1.33-73
< #DISTFILES=	anki-${DISTVERSION}-node_modules.tgz:npmcache
< DISTFILES=	anki-2.1.33-node_modules.tgz:npmcache
> DISTFILES=	anki-${DISTVERSION}-node_modules.tgz:npmcache
< _MY_BUILDHASH=	340c749d
> _MY_BUILDHASH=	84dcaa8

I ran into two problems. When compiling a clean build, I get a failure at:

    Command failed: npm --silent install chalk@^4.0.0 jsdoc@^3.6.3 tmp@^0.2.0 uglify-js@^3.7.7 espree@^7.0.0 escodegen@^1.13.0 estraverse@^5.1.0
    gmake[3]: *** [Makefile:27: .build/proto] Error 1
    gmake[3]: Leaving directory '/usr/obj/usr/ports/games/anki/work/anki-2.1.35/ts'
    gmake[2]: *** [Makefile:119: build] Error 2
    gmake[2]: Leaving directory '/usr/obj/usr/ports/games/anki/work/anki-2.1.35'
    ===> Compilation failed unexpectedly.

However, I can manually cd into /usr/obj/usr/ports/games/anki/work/anki-2.1.35/ts and run the npm command (minus the '--silent' option). The npm install succeeds (no unusual output, '0' exit code).

If I cd back to /usr/ports/games/anki, I can then continue compilation, which now succeeds.

However, when installing I get an error about missing files:

    pkg-static: Unable to access file /usr/obj/usr/ports/games/anki/work/stage/usr/local/share/anki/aqt_data/web/congrats.html:No such file or directory
    pkg-static: Unable to access file /usr/obj/usr/ports/games/anki/work/stage/usr/local/share/anki/aqt_data/web/congrats.js:No such file or directory
    *** Error code 74

I'm not sure where the actual problem is, but I removed both 'congrats.html' and 'congrats.js' from '/usr/obj/usr/ports/games/anki/work/.PLIST.mktmp' and from 'pkg-plist', and was then able to install. Anki seems to work fine, syncing, card creation, review, etc.
Comment 8 Kai Knoblich freebsd_committer 2020-10-26 13:52:06 UTC
Created attachment 219101 [details]

(In reply to David Schlachter from comment #7)

Thank you for the info which was very helpful to track down those issues and the additional testing.

Attached is an updated patch for the 2.1.35 release of Anki and it also contains an improved "make-npm-cache" target that really generates a npm cache (= ${WRKSRC}/.npm) and not a pre-generated "node_modules" directory.

The issue with the "npx pbjs" command was caused by additional dependencies that are required by the CLI component of the "protobufjs" package. Those packages couldn't be installed during the build phase due the environment variable "npm_config_offline=true" which prevents "npm" to open active network connections.

The template file "congrats.html" (from which "congrats.js" is generated) isn't included in the 2.1.35 release for some reason as far I can tell.

It seems that the buildhash needs to be exactly 8 digits otherwise the check for new versions always reports that a newer version is available.
Comment 9 Kai Knoblich freebsd_committer 2020-10-26 13:53:58 UTC
Created attachment 219102 [details]

Slightly improved patch for devel/py-maturin (fixes in pkg-descr and the handling of concurrent installations).