Created attachment 261795 [details] www/iocaine: add new port (version 2.4.1) Iocaine is a defense mechanism against aggressive AI crawlers (which pretty much all ignore robots.txt). It does not fend off those bots, but traps them in an endless maze of plausibly looking garbage to poison them. The port was made with a lot of help by Yusuf Yaman (nxjoseph@protonmail.com). Credits to him for providing a patch against src/bin/iocaine.rs and an initial working Makefile as a starting point for this port. Also thanks to the upstream maintainer Gergely Nagy (algernon) for patching 2 build errors due to linuxisms and coming forward to test build future releases on FreeBSD. For reference and history (first attempts were made with version 2.2.0) the thread on the freebsd forums: https://forums.freebsd.org/threads/configure-phase-fails-because-file-with-same-name-as-wrksrc-cannot-be-created.97837 And the upstream PR: https://git.madhouse-project.org/iocaine/iocaine/issues/31 The port was tested with 'poudriere testport' on 14.3-RELEASE and finished without errors; portlint did not show any errors or warnings (apart from 'Consider to set DEVELOPER=yes in /etc/make.conf') and the built package (via 'poudriere bulk') installs & runs fine on 14.3-RELEASE.
Created attachment 261796 [details] www/iocaine: add new port (version 2.4.1) fixed 2 trailing whitespaces that slipped through
Created attachment 261797 [details] www/iocaine: add new port (version 2.4.1) upstream tarball size changed after CI fix (really sorry for yet another bump)
Created attachment 262336 [details] www/iocaine: add new port (version 2.5.0) *bump* updated patch to current version 2.5.0
Is there a reason why this port, newly submitted on 2025-07-01 13:33:25 UTC, is not present in the ports tree yet (2026-01-07 16:06 UTC) ?
Curiousity: How does this differ from Anubis?
(In reply to void from comment #4) I don't know why it wasn't commited to the ports tree yet - I guess committers are rather busy, so I didn't want to become too annoying. That being said: Version 2.5.0 is outdated by now anyways. The new 3.x branch got an overhaul of the configuration and scripting engine, so it is not a drop-in upgrade. Version 3.1 builds and runs successfully and I already began to update the port, but I haven't come around to review and change the included config files/templates and update the pkg-message to reflect all those changes. TBH I also didn't put that very high on my priority list since this PR didn't get any attention for almost half a year anyways... If there is interest to commit this to ports, I'll update it to version 3.1 in the next few days. (In reply to Michael Osipov from comment #5) Anubis is an (IMHO) annoying 'are you a robot' challange that gets in the way of everyone and significantly slows down the browsing experience for humans. If a browser is considered a bot, it is simply not served any content. Iocaine specifically targets AI crawlers and sends them into an endless maze of legitimately looking garbage. So it is basically a tarpit to poison AI models. It can be either used with very crude and simple decision models, e.g. by checking the http_user_agent string at the webserver/reverse proxy, or one can leverage its scripting engine to make detection more advanced. It *can* also use some form of proof-of-work or other captchas, but as the documentation/description of the project also states: this form of decision should always be the last resort as it is getting in the way of everyone and is massively annoying to humans, especially if you are using a browser that by default blocks cookies and javascript. I personally use iocaine with the simple http_user_agent check (using the list from www/github.com/ai-robots-txt/ai-robots.txt) at my webservers and reverse proxies. Clients matching those strings can still retreive the robots.txt, which contains a strict 'disallow' rule telling them to sod off - if they ignore it and try to access anything else, they get redirected to iocaine and are fed with gabrage. From my logs it turns out they will happily eat several GB of crap until they settle down and disappear completely after a while. Occasionally they reappear but then cut out very quickly. In raw numbers from my experience on several hosts: after ~2 months attacks from the most aggressive crawlers like OpenAI/ChatGPT, meta and Google have almost vanished completely. If they return, they usually stop after 2 requests and at a "maze depth" of 1-2. Total traffic went down by >50% (up to 80% on some hosts!) and I haven't seen a single occurence of a host being choked ever since. Iocaine itself is extremely lightweigt, so apart from bandwidth (which can be easily limited), it adds almost no cost and reduces load on backend servers very significantly.
(In reply to Sebastian Oswald from comment #6) Thank you for the explanation! It is much more than I have expected.
Just wanted to add that I would also be interested in this port getting updated and committed. Been looking at Anubis instead, as it’s also starting to add similar functionality, but iocaine is just way more advanced in this way, so I‘d prefer to use is.
(In reply to Sebastian Oswald from comment #6) >If there is interest to commit this to ports, I'll update it to >version 3.1 in the >next few days. I am very interested and would be very grateful if you'd please add the updated port.
Thanks for the feedback. I'll update the port within the next few days. I Just recognized that it won't build via poudriere, because since version 3.1.0 it pulls additional files during build, so I have to modify the Makefile somewhat more than expected, but it should be solvable.
Created attachment 267114 [details] www/iocaine: add new port (version 3.1.0) I updated the port to reflect all changes made to the configuration for version 3.x poudriere testport & bulk runs fine for me without any errors; resulting pkg installs fine. (all @14.3-RELEASE /w latest ports tree) The service should start out-of-the-box after installation, one only needs to point a webserver/reverse-proxy on the same host to 127.0.0.1:42069. It would be nice to get some feedback before this gets committed, especially if something seems to be broken or needs improvement. Thanks!
Created attachment 267123 [details] www/iocaine: add new port (version 3.1.0) For whatever reason the checksum of robots.json changed; I updated the patch.
Patch is good. I refused to wait any longer for this port to hit the tree and applied this to my local branch so I could use it now. Works as intended. Please commit! @Sebastian Thanks for all your work and *patients* on this. I have no idea why this port has been overlooked. Perhaps a request for a committer on the ports@ mailing list will get this the attention it needs. Thanks!
(In reply to Chris Hutchinson from comment #13) Thanks for the feedback! I'll send a request to ports@ and see if someone finds the time to commit this.
(In reply to Sebastian Oswald from comment #14) testbuilds@work
(In reply to void from comment #4) yes, no free time from ports committers 8-(
(In reply to Kurt Jaeger from comment #16) and it turns out, it uses rust, which leads to a longer compile cycle if one tries to compile it on 16/15/14.3/13.5, because all of those jails need to build rust 8-(
(In reply to Kurt Jaeger from comment #17) This is mildly put. In reality this is PITA even on a 20-core Xeon. The entire build for the package takes more than an hour.
(In reply to Michael Osipov from comment #18) my 32core amd box needs approx. 4.5h per rust build, so testing this port will take approx. 18 hours wall-clock only to testbuild 8-( While in general using rust is useful, the practical issues can not be ignored 8-(
(In reply to Kurt Jaeger from comment #19) BTW, the packaging of Rust takes abysmally long because DOCS and SOURCES are included and the amount of small files is huge.
(In reply to Kurt Jaeger from comment #19) I was lucky, only one rust was build. But: all four failed to build ? https://people.freebsd.org/~pi/logs/ 135-iocaine-3.1.0.log 143-iocaine-3.1.0.log 150-iocaine-3.1.0.log 160-iocaine-3.1.0.log Any ideas ?
(In reply to Kurt Jaeger from comment #21) > The system library `libzstd` required by crate `zstd-sys` was not found. > The file `libzstd.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory. > The PKG_CONFIG_PATH environment variable is not set. This sadly is a bug with the bundled LLVM which still hasn't been fixed by rust: https://github.com/rust-lang/rust/issues/135503 I completely forgot about that, but now that I saw your log entry I remembered running into this compile error before (not with iocaine but with other rust-based ports). The solution is to build rust with PORT_LLVM=on so it looks at the correct paths for system libraries. The LLVM version bundled with rust omits /usr/local/lib, so this problem exists for any OS adhering to the unix hierarchy and puts third-party libraries in /usr/local/lib (linux does not, so it works for them). I've set lang_rust_SET+=PORT_LLVM in my makefiles a long time ago when first running into that problem and completely forgot about it. (and disabled DOCS and SOURCES for that matter) And yes, building rust is a major PITA, especially its insane memory/tempfs requirements (which are considered "OK" and "normal" by its devs...). I have to build it for other ports like firefox, so having another port that requires is doesn't hurt as much. Building the iocaine port itself should take well under 15 minutes (~13 minutes on my old Xeon E5-2660v4 based buildhost) I'm not sure if we should bring up that compile error with lang/rust, e.g. to default to PORT_LLVM=on, or urge upstream to finally fix that bug (which I guess is pointless...)
Built fine on E5-2690 v2 dual xeon (sht turned off so 20 cores) Took 16mins 26sec Log is at https://www.f451.online/~void/FreeBSD/poudriere/iocaine-3.1.0.log PORT_LLVM is *not* enbled in rust options. DOCS is though. And this is ports-mgmt/poudriere-dsh2dsh rather than straight poudriere or -devel, so rust doesn't get rebuilt nearly as often.
(In reply to void from comment #23) 16 min on such a machine is massive IMHO.
(In reply to Michael Osipov from comment #24) Most things requiring rust do take ages though, so... it is what it is
What is preventing this port from being committed to the ports tree?
(In reply to void from comment #26) my testbuilds failed. Are there changes, should I try again ?
(In reply to Kurt Jaeger from comment #27) As I already stated: PORT_LLVM in lang/rust needs to be enabled as their bundled LLVM is (still!) broken and ignores local libraries (in this case: libzstd).
(In reply to Sebastian Oswald from comment #22) ah, thanks, now I remember (two months 8-)). I think the package-builder boxes do not have PORT_LLVM set, so the package would not be available. I have no real solution for this right now.
Assign to nxjoseph@, who aided with creating the port in the first place.
My and64 environment is somewhat older for both main [16] and ports but I tried with the Makefile having: USES= cargo gmake pkgconfig and using poudriere-devel: [00:00:05] [09] [00:00:00] Building www/iocaine | iocaine-3.1.0 [00:07:37] [09] [00:07:32] Finished www/iocaine | iocaine-3.1.0: Success I do not use PORT_LLVM for building lang/rust (never have). May be other folks should test using: USES= cargo gmake pkgconfig For reference: # uname -apKU FreeBSD 7950X3D-ZFS 16.0-CURRENT FreeBSD 16.0-CURRENT main-n282840-4b0d5d1d6a7c GENERIC-NODEBUG amd64 amd64 1600007 1600007 # ~/fbsd-based-on-what-commit.sh -C /usr/ports 87b74af44b25 (HEAD -> main, freebsd/main, freebsd/HEAD) print/plutobook: Update to 0.12.0 Author: Joel Bodenmann <jbo@FreeBSD.org> Commit: Joel Bodenmann <jbo@FreeBSD.org> CommitDate: 2025-12-23 22:59:57 +0000 branch: main merge-base: 87b74af44b25bf283df0237a77e763b01b38fa84 merge-base: CommitDate: 2025-12-23 22:59:57 +0000 n730216 (--first-parent --count for merge-base) Just for illustration (from the live environment, not poudriere-devel's jail in operation): # pkg which /usr/local/libdata/pkgconfig/libzstd.pc /usr/local/libdata/pkgconfig/libzstd.pc was installed by package zstd-1.5.7_1 The analogous in the bulk run's jail was automatically used because the path /usr/local/libdata/pkgconfig/ was set up to automatically be searched because of USES= pkgconfig being involved. (At least, that is my understanding.) The actual error messsage logs that had been reported for the failure reported the likes of (one example): [zstd-sys 2.0.16+zstd.1.5.7] The system library `libzstd` required by crate `zstd-sys` was not found. [zstd-sys 2.0.16+zstd.1.5.7] The file `libzstd.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory. [zstd-sys 2.0.16+zstd.1.5.7] The PKG_CONFIG_PATH environment variable is not set. [zstd-sys 2.0.16+zstd.1.5.7] [zstd-sys 2.0.16+zstd.1.5.7] HINT: if you have installed the library, try setting PKG_CONFIG_PATH to the directory containing `libzstd.pc`.
The port built in 15 mins 10s for me using [00:00:00] Poudriere version: poudriere-git-3.4.99.20260304 [00:00:00] Host OSVERSION: 1500504 [00:00:00] Jail OSVERSION: 1600013 The poudriere jail was freshly built and the ports tree was from a few hrs ago from the date of this post. The port applied was from https://bugs.freebsd.org/bugzilla/attachment.cgi?id=267123&action=diff in comment 12. No options were selected. No modifications to the makefile of hte port were needed.
Created attachment 268864 [details] diff for some adjustments Hi. 1. files/pkg-message.in is not formatted in UCL Format. https://docs.freebsd.org/en/books/porters-handbook/book/#porting-message 2. iocaine requires /var/db/iocaine directory to exist and owned by appropriate users/groups. 3. Instead of @${MV} ${DISTDIR} ... Use @${CP} ${DISTDIR} ... since it failed when Poudriere build is run as user and this is more appropriate because it doesn't move (delete) assets from distfiles. 4. `stage-qa` encourages to add libzstd.so:archivers/zstd to LIB_DEPENDS. 5. `portlint -AC` discourages adding PORTREVISION= to new ports. 5. The port may be polished by portfmt, portlint and portclippy. 6. I have moved in-makefile plist entries to file: pkg-plist since `portlint` also warns about using %%FOO%% in PLIST_FILES. 7. Remove ${DOCSDIR} directory creation since it is unused. All the adjustments above is contained in my attached diff. I also would like to mention that current version v3.1.0 is like 2 months old. Thank you.
(In reply to Sebastian Oswald from comment #28) PORT_LLVM was *not* set in my build on 15-stable on 2026-01-14 16:57:49 UTC or on -current on 2026-03-16 21:53:41 UTC in comment 32 using https://bugs.freebsd.org/bugzilla/attachment.cgi?id=267123&action=diff from comment 12
Using the latest patch https://bugs.freebsd.org/bugzilla/attachment.cgi?id=268864 from comment 23, this port builds fine across all versions. Here are the logs: 13.5: https://www.f451.online/~void/FreeBSD/poudriere/iocaine-2026-03-17/135-iocaine-3.1.0.log 14.4: https://www.f451.online/~void/FreeBSD/poudriere/iocaine-2026-03-17/144-iocaine-3.1.0.log 15.0: https://www.f451.online/~void/FreeBSD/poudriere/iocaine-2026-03-17/150-iocaine-3.1.0.log 16.0-current: https://www.f451.online/~void/FreeBSD/poudriere/iocaine-2026-03-17/current-iocaine-3.1.0.log
What have folks been doing that might lead to differences in what curl depends on? It appears that lang/rust 's curl dependency differences may be the important difference between the accessible failing logs from back in Jan and the new working logs that are now accessible. A failing example from comment #21 references has: ===> Installing existing package /packages/All/rust-1.92.0.pkg [160-default-job-01] Installing rust-1.92.0... [160-default-job-01] `-- Installing curl-8.17.0... [160-default-job-01] | `-- Installing brotli-1.2.0,1... [160-default-job-01] | `-- Extracting brotli-1.2.0,1: .......... done [160-default-job-01] | `-- Installing libidn2-2.3.8... [160-default-job-01] | | `-- Installing indexinfo-0.3.1_1... [160-default-job-01] | | `-- Extracting indexinfo-0.3.1_1: . done [160-default-job-01] | | `-- Installing libunistring-1.4.1... [160-default-job-01] | | `-- Extracting libunistring-1.4.1: .......... done [160-default-job-01] | `-- Extracting libidn2-2.3.8: .......... done [160-default-job-01] | `-- Installing libnghttp2-1.68.0... [160-default-job-01] | `-- Extracting libnghttp2-1.68.0: ....... done [160-default-job-01] | `-- Installing libssh2-1.11.1,3... [160-default-job-01] | `-- Extracting libssh2-1.11.1,3: .......... done [160-default-job-01] | `-- Installing openldap26-client-2.6.10_1... [160-default-job-01] | | `-- Installing cyrus-sasl-2.1.28_5... *** Added group `cyrus' (id 60) *** Added user `cyrus' (id 60) [160-default-job-01] | | `-- Extracting cyrus-sasl-2.1.28_5: .......... done [160-default-job-01] | `-- Extracting openldap26-client-2.6.10_1: .......... done [160-default-job-01] `-- Extracting curl-8.17.0: .......... done [160-default-job-01] Extracting rust-1.92.0: .......... done . . . ===> iocaine-3.1.0 depends on package: rust>=1.92.0 - found . . . =======================<phase: lib-depends >============================ ===== env: DEVELOPER_MODE=yes USE_PACKAGE_DEPENDS_ONLY=1 USER=root UID=0 GID=0 ===> iocaine-3.1.0 depends on shared library: libcurl.so - found (/usr/local/lib/libcurl.so) =========================================================================== Comparing to a working example (later below), note the rust-1.92.0 and curl installation's lack of anything like: [00:02:25] [redacted.home.arpa] | `-- Installing zstd-1.5.7_1... [00:02:25] [redacted.home.arpa] | `-- Extracting zstd-1.5.7_1: .......... done and the missing status of anything analogous the working example's lib-depends line: [00:02:31] ===> iocaine-3.1.0 depends on shared library: libzstd.so - found (/usr/local/lib/libzstd.so) A working example from comment #35 references: [00:02:21] ===> Installing existing package /packages/All/rust-1.93.1.pkg [00:02:22] [redacted.home.arpa] Installing rust-1.93.1... [00:02:22] [redacted.home.arpa] `-- Installing curl-8.17.0... [00:02:22] [redacted.home.arpa] | `-- Installing brotli-1.2.0,1... [00:02:22] [redacted.home.arpa] | `-- Extracting brotli-1.2.0,1: .......... done [00:02:22] [redacted.home.arpa] | `-- Installing libidn2-2.3.8... [00:02:22] [redacted.home.arpa] | | `-- Installing indexinfo-0.3.1_1... [00:02:22] [redacted.home.arpa] | | `-- Extracting indexinfo-0.3.1_1: .... done [00:02:22] [redacted.home.arpa] | | `-- Installing libunistring-1.4.2... [00:02:22] [redacted.home.arpa] | | `-- Extracting libunistring-1.4.2: .......... done [00:02:24] [redacted.home.arpa] | `-- Extracting libidn2-2.3.8: .......... done [00:02:24] [redacted.home.arpa] | `-- Installing libnghttp2-1.68.0... [00:02:24] [redacted.home.arpa] | `-- Extracting libnghttp2-1.68.0: .......... done [00:02:25] [redacted.home.arpa] | `-- Installing libpsl-0.21.5_2... [00:02:25] [redacted.home.arpa] | `-- Extracting libpsl-0.21.5_2: .......... done [00:02:25] [redacted.home.arpa] | `-- Installing libssh2-1.11.1,3... [00:02:25] [redacted.home.arpa] | `-- Extracting libssh2-1.11.1,3: .......... done [00:02:25] [redacted.home.arpa] | `-- Installing zstd-1.5.7_1... [00:02:25] [redacted.home.arpa] | `-- Extracting zstd-1.5.7_1: .......... done [00:02:25] [redacted.home.arpa] `-- Extracting curl-8.17.0: .......... done [00:02:26] [redacted.home.arpa] Extracting rust-1.93.1: .......... done [00:02:30] ===> iocaine-3.1.0 depends on package: rust>=1.93.0 - found . . . =======================<phase: lib-depends >============================ [00:02:30] ===== env: USE_PACKAGE_DEPENDS_ONLY=1 USER=root UID=0 GID=0 [00:02:30] ===> iocaine-3.1.0 depends on shared library: libcurl.so - found (/usr/local/lib/libcurl.so) [00:02:31] ===> iocaine-3.1.0 depends on shared library: libzstd.so - found (/usr/local/lib/libzstd.so) [00:02:31] =========================================================================== Comparing/contrasting in the other direction: Note that the earlier failing one has: [160-default-job-01] | `-- Installing openldap26-client-2.6.10_1... [160-default-job-01] | | `-- Installing cyrus-sasl-2.1.28_5... *** Added group `cyrus' (id 60) *** Added user `cyrus' (id 60) [160-default-job-01] | | `-- Extracting cyrus-sasl-2.1.28_5: .......... done [160-default-job-01] | `-- Extracting openldap26-client-2.6.10_1: .......... done for which the working one has nothing analogous. What lead to this openldap26-client / cyrus-sasl vs. zstd difference in the builds?
(In reply to Mark Millard from comment #36) The following shows that the openldap26-client / cyrus-sasl vs. zstd difference was truth back in the reported failure time frame as well . . . I'll note that comment #23 reference to a successful build of the same time frame as #21 has: [00:01:35] [redacted.home.arpa] | `-- Installing zstd-1.5.7_1... [00:01:35] [redacted.home.arpa] | `-- Extracting zstd-1.5.7_1: .......... done and is based on rust 1.92.0 (like the failure examples) --but does not have an equivalent of the: [00:02:31] ===> iocaine-3.1.0 depends on shared library: libzstd.so - found (/usr/local/lib/libzstd.so) For reference: [00:01:34] [redacted.home.arpa] Installing rust-1.92.0... [00:01:34] [redacted.home.arpa] `-- Installing curl-8.17.0... [00:01:34] [redacted.home.arpa] | `-- Installing brotli-1.2.0,1... [00:01:34] [redacted.home.arpa] | `-- Extracting brotli-1.2.0,1: .......... done [00:01:34] [redacted.home.arpa] | `-- Installing libidn2-2.3.8... [00:01:34] [redacted.home.arpa] | | `-- Installing indexinfo-0.3.1_1... [00:01:34] [redacted.home.arpa] | | `-- Extracting indexinfo-0.3.1_1: .... done [00:01:34] [redacted.home.arpa] | | `-- Installing libunistring-1.4.1... [00:01:34] [redacted.home.arpa] | | `-- Extracting libunistring-1.4.1: .......... done [00:01:34] [redacted.home.arpa] | `-- Extracting libidn2-2.3.8: .......... done [00:01:35] [redacted.home.arpa] | `-- Installing libnghttp2-1.68.0... [00:01:35] [redacted.home.arpa] | `-- Extracting libnghttp2-1.68.0: .......... done [00:01:35] [redacted.home.arpa] | `-- Installing libpsl-0.21.5_2... [00:01:35] [redacted.home.arpa] | `-- Extracting libpsl-0.21.5_2: .......... done [00:01:35] [redacted.home.arpa] | `-- Installing libssh2-1.11.1,3... [00:01:35] [redacted.home.arpa] | `-- Extracting libssh2-1.11.1,3: .......... done [00:01:35] [redacted.home.arpa] | `-- Installing zstd-1.5.7_1... [00:01:35] [redacted.home.arpa] | `-- Extracting zstd-1.5.7_1: .......... done [00:01:35] [redacted.home.arpa] `-- Extracting curl-8.17.0: .......... done [00:01:36] [redacted.home.arpa] Extracting rust-1.92.0: .......... done . . . [00:01:39] =======================<phase: lib-depends >============================ [00:01:39] ===== env: USE_PACKAGE_DEPENDS_ONLY=1 USER=root UID=0 GID=0 [00:01:40] ===> iocaine-3.1.0 depends on shared library: libcurl.so - found (/usr/local/lib/libcurl.so) [00:01:40] =========================================================================== So again: What leads to this openldap26-client / cyrus-sasl vs. zstd difference in the builds?
(In reply to Yusuf Yaman from comment #33) Should things work by finding and using libdata/pkgconfig/libzstd.pc ? The question is promoted by the messages: [zstd-sys 2.0.16+zstd.1.5.7] The system library `libzstd` required by crate `zstd-sys` was not found. [zstd-sys 2.0.16+zstd.1.5.7] The file `libzstd.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory. [zstd-sys 2.0.16+zstd.1.5.7] The PKG_CONFIG_PATH environment variable is not set. As I understand, having pkgconfig in USES would make the ports infrastructure find and use libdata/pkgconfig/libzstd.pc automatically at the appropriate point (instead of reporting those messages). (But I'm no expert in the area.)
Created attachment 268908 [details] iocaine-3.1.0.log.gz (In reply to Mark Millard from comment #38) > Should things work by finding and using libdata/pkgconfig/libzstd.pc ?? I am unable to reproduce this problem. > The question is promoted by the messages: > [zstd-sys 2.0.16+zstd.1.5.7] The system library `libzstd` required by crate > `zstd-sys` was not found. > [zstd-sys 2.0.16+zstd.1.5.7] The file `libzstd.pc` needs to be installed and > the PKG_CONFIG_PATH environment variable must contain its parent directory. > [zstd-sys 2.0.16+zstd.1.5.7] The PKG_CONFIG_PATH environment variable is not set. I don't see these messages in my poudriere log. I keep the USES as is. $ grep USES Makefile USES= cargo gmake I have attached my iocaine poudriere log that is compressed with gzip. Thanks.
(In reply to Yusuf Yaman from comment #39) One needs to test a context where curl does not lead to zstd being installed in order to see a failure. See comment #36 and #37 . The only failure logs referenced indicate a lack of zstd from the curl install; the ones referenced for success have all had zstd installed via the curl installation. That includes the .log.gz that you have supplied that is for a success: ===> Installing existing package /packages/All/rust-1.93.1.pkg [143amd64-default-job-01] Installing rust-1.93.1... [143amd64-default-job-01] `-- Installing curl-8.17.0... [143amd64-default-job-01] | `-- Installing brotli-1.2.0,1... [143amd64-default-job-01] | `-- Extracting brotli-1.2.0,1: .......... done [143amd64-default-job-01] | `-- Installing libidn2-2.3.8... [143amd64-default-job-01] | | `-- Installing indexinfo-0.3.1_1... [143amd64-default-job-01] | | `-- Extracting indexinfo-0.3.1_1: .... done [143amd64-default-job-01] | | `-- Installing libunistring-1.4.2... [143amd64-default-job-01] | | `-- Extracting libunistring-1.4.2: .......... done [143amd64-default-job-01] | `-- Extracting libidn2-2.3.8: .......... done [143amd64-default-job-01] | `-- Installing libnghttp2-1.68.0... [143amd64-default-job-01] | `-- Extracting libnghttp2-1.68.0: .......... done [143amd64-default-job-01] | `-- Installing libpsl-0.21.5_2... [143amd64-default-job-01] | `-- Extracting libpsl-0.21.5_2: .......... done [143amd64-default-job-01] | `-- Installing libssh2-1.11.1,3... [143amd64-default-job-01] | `-- Extracting libssh2-1.11.1,3: .......... done [143amd64-default-job-01] | `-- Installing zstd-1.5.7_1... [143amd64-default-job-01] | `-- Extracting zstd-1.5.7_1: .......... done [143amd64-default-job-01] `-- Extracting curl-8.17.0: .......... done [143amd64-default-job-01] Extracting rust-1.93.1: .......... done ===> iocaine-3.1.0 depends on package: rust>=1.93.0 - found As far as I can tell, as long as the curl installation leads to: [143amd64-default-job-01] | `-- Installing zstd-1.5.7_1... [143amd64-default-job-01] | `-- Extracting zstd-1.5.7_1: .......... done the reported failure is not going to happen no matter what the other details are. Testing needs to be without those lines happening. If you can, retest with a context where the curl installation does not cause zstd to be installed. curl has a ZSTD option that can be disabled.
(In reply to Mark Millard from comment #40) Hmmm, I see. Then I guess it can be fixed by adding archivers/zstd to LIB_DEPENDS? From my comment #33: > 4. `stage-qa` encourages to add libzstd.so:archivers/zstd to LIB_DEPENDS. Sorry, I am not willing to rebuild rust for this option change. [00:00:02] [Dry Run] Deleting curl-8.17.0.pkg: changed options [00:00:02] [Dry Run] Pkg: +ALTSVC +BROTLI -CARES +COOKIES -CURL_DEBUG -DEBUG +DICT +DOCS +EXAMPLES +FTP -GNUTLS +GOPHER -G SSAPI_MIT +GSSAPI_NONE +HTTP +HTTP2 +IDN +IMAP +IPFS +IPV6 -LDAP -LDAPS -LIBSSH +LIBSSH2 -LIBUV +MQTT +NTLM +OPENSSL +POP3 +PROXY +PSL +RTSP +SMB +SMTP +STATIC +TELNET +TFTP +THREADED_RESOLVER +TLS_SRP +WEBSOCKET -WOLFSSL +ZSTD [00:00:02] [Dry Run] New: +ALTSVC +BROTLI -CARES +COOKIES -CURL_DEBUG -DEBUG +DICT +DOCS +EXAMPLES +FTP -GNUTLS +GOPHER -G SSAPI_MIT +GSSAPI_NONE +HTTP +HTTP2 +IDN +IMAP +IPFS +IPV6 -LDAP -LDAPS -LIBSSH +LIBSSH2 -LIBUV +MQTT +NTLM +OPENSSL +POP3 +PROXY +PSL +RTSP +SMB +SMTP +STATIC +TELNET +TFTP +THREADED_RESOLVER +TLS_SRP +WEBSOCKET -WOLFSSL -ZSTD [00:00:04] [Dry Run] Checking packages for missing dependencies [00:00:04] [Dry Run] Deleting iocaine-3.1.0.pkg: missing dependency: curl-8.17.0 [00:00:04] [Dry Run] Deleting rust-1.93.1.pkg: missing dependency: curl-8.17.0 Thanks.
(In reply to Yusuf Yaman from comment #41) I'll note that your: +LIB_DEPENDS= libcurl.so:ftp/curl \ + libzstd.so:archivers/zstd looks to probably be fine and my "pkgconfig in USES" related material in comment #38 should likely be ignored. Hopefully, someone with a context where curl does not install zstd will test your "diff for come adjustments" to be sure that it works for such a context.
(In reply to Mark Millard from comment #42) > I'll note that your: > +LIB_DEPENDS= libcurl.so:ftp/curl \ > + libzstd.so:archivers/zstd > looks to probably be fine and my "pkgconfig in USES" > related material in comment #38 should likely be > ignored. > Hopefully, someone with a context where curl does not > install zstd will test your "diff for some adjustments" > to be sure that it works for such a context. I see, okay. I will do the rebuild and see. Thanks.
I have lang/rust built without ZSTD option and yes iocaine fails to build in this case. Adding libzstd to LIB_DEPENDS seems to solve it. I wonder if iocaine needs curl or is libzstd enough?
So, what I learn here is that having something like: +CARGO_CRATES= adler2-2.0.1 \ . . . + zstd-sys-2.0.16+zstd.1.5.7 and: diff --git a/www/iocaine/distinfo b/www/iocaine/distinfo new file mode 100644 index 00000000000..f0405a2b90d --- /dev/null +++ b/www/iocaine/distinfo @@ -0,0 +1,995 @@ . . . +SHA256 (rust/crates/zstd-sys-2.0.16+zstd.1.5.7.crate) = 91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748 +SIZE (rust/crates/zstd-sys-2.0.16+zstd.1.5.7.crate) = 775620 is not enough to tell the ports infrastructure that zstd-sys needs the libzstd.so:archivers/zstd as a ports based dependency. Once that is done separately in the Makefile, lang/rust does not need changes for this to avoid the documented build failures, even when rust installing curl does not in turn install zstd as well. Of course, "diff for some adjustments" covers more than just adding the libzstd.so:archivers/zstd dependency and likely should be used. My investigation/analysis has been specific to the one issue.
(In reply to Yusuf Yaman from comment #44) Yes, this looks workable. I did testbuilds, all fine excep 13.5...
(In reply to Kurt Jaeger from comment #46) > Yes, this looks workable. I did testbuilds, all fine excep 13.5... Thanks for testing. Did it fail on 13.5 or didn't test on it? I only have 14.3 poudriere jail :/
(In reply to Yusuf Yaman from comment #47) I test-build with curl on 8.19.0, and this triggered this: [00:00:03] Ignoring ftp/curl | curl-8.19.0: only supports LDAPS with GnuTLS/OpenSSL/wolfSSL [00:00:03] Skipping www/iocaine | iocaine-3.1.0: Dependent port ftp/curl | curl-8.19.0 ignored [00:00:03] Skipping lang/rust | rust-1.93.1: Dependent port ftp/curl | curl-8.19.0 ignored We should probably ignore this, because 13.5 will be EOL at April 30, 2026.
Sorry for chiming in so late, I got knocked out by a cold the whole of last week and just started to catch up on stuff yesterday. I'm currently battling a regression introduced by review D49967 which inflates build times by a factor of 3-4 on 14.4-RELEASE. I.e., building rust now takes over 2h on my build hosts instead of ~35-40 minutes... The alleged fix of using WITH_LLVM_STATIC (as introduced by review D50956) does not work - the binaries for llvm, clang et al are back to normal sizes in the poudriere jail, but bulk/build times are still as bad, so I'm currently in the process of completely ripping out the changes of those two commits. Since this regression is still present on the hosts (both have been updated to 14.4-RELEASE), buildworld of course is also affected and also takes *A LOT* longer than usual and makes this process a major PITA... As soon as I have one of my buildhosts running at normal build speeds again, I will also give the diff from comment #33 by @nxjoseph a try. Thanks to everyone working on this and especially to Yusuf, and apologies for causing you addditional workload.
(In reply to Sebastian Oswald from comment #49) Are you using WITH_LLVM_LINK_STATIC_LIBRARIES ? review D50956 had: Diff 6 168499 823ebd7 * Rename option to `WITH_LLVM_LINK_STATIC_LIBRARIES`
(In reply to Mark Millard from comment #50) Yes, the very first thing I tried for both hosts was building world for the poudriere jail with 'WITH_LLVM_LINK_STATIC_LIBRARIES' enabled, and as said this increases clang/llvm binaries back to ther original size. But package build times for some reason were still just as bad as before (llvm19 1h58min, rust 2h07min), so either it still uses the external lib for some reason, or the tunable is simply broken/buggy some other way for 14.4-RELEASE. However, build world with a patch that completely reverts D49967 and D50956 finally finished yesterday evening and I started a bulk build over night. llvm19 was back down to 20 minutes (was ~18-26min for bulk builds @14.3-RELEASE), rust is down to 42 minutes (36-50min @14.3-RELEASE), so everything is back to normal and I can bulk build within reasonable timeframes again. (I really feel the urge to rant about how such a regression was able/allowed to make it into a release, but this PR isn't the place for that...) SO ANYHOW... I'm currently catching up on building my pkg sets from the latest portst tree, including iocaine with Yusufs latest diff. (the iocaine port already got successfully built) As soon as the bulk job is finished, I'll update the installations i'm running from that resulting package, so I can verify everything works as intended. Regarding that libzstd problem, I couldn't really follow if ZSTD=OFF in rust or curl is causing the build failure, so I prepared a poudriere set with a global ZSTD=off (but otherwise vanilla port options and makefile) to test for those build errors or respectively to verify that the LIB_DEPENDS fix by Yusuf works. I hope I can kick off those builds around late afternoon today, right after the slightly more urgent bulk builds of the package sets for my/our production servers are finished...
(In reply to Sebastian Oswald from comment #51) > . . . (llvm19 1h58min, rust 2h07min) . . . > llvm19 was back down to 20 minutes (was ~18-26min for bulk builds @14.3-RELEASE), > rust is down to 42 minutes (36-50min @14.3-RELEASE) ) Are these llvm19 times not counting building prerequisites/dependencies first? Similarly: rust? ) Was only 1 builder active during the llvm19 build that was timed? During the rust build that was timed? (No parallel building jobs during those?) ) Was the system clang/clang++ used to build rust ? Some devel/llvm* ? ) Were "disabled DOCS and SOURCES" disabled for rust ? ) Any other tailoring of the rust build? Of the llvm19 build? ) USE_TMPFS=??? Any use of blacklisting tmpfs use? ) . . . Basically: information for helping replication testing but not of detailed times across likely differing systems and possible other configuration differences) llvm19 had the larger ratio (118min/20/min) and so is likely better as a test if only one is tried.
(In reply to Sebastian Oswald from comment #51) A 14.4 vs. 14.3 comparison/contrast example, lang/llvm22 based Just my personal environment, so building lang/llvm22 instead lang/llvm19 nondebug boot kernel and debug boot world WITH_LLVM_LINK_STATIC_LIBRARIES= in use repository initially empty. 7950X3D 16 SMT cores, so 32 freebsd cpus, all used. 192 GiBytes of RAM. ZFS. 1.4T PCIe based Optane media. Only the 1 builder was active for the below building ... finished pairs. 14.4-RELEASE official pkgbase jail-world installation: (a manual line wrap) [00:00:01] Inspecting /usr/local/poudriere/data/.m/release14p3-amd64-alt/ref//usr/ports for modifications to git checkout... no [00:00:02] Ports top-level git hash: 942c4a21a22ead9e98c150a9388b2eba71a75af2 . . . [00:06:25] [01] [00:00:00] Building devel/llvm22@default | llvm22-22.1.2 [00:31:41] [01] [00:25:16] Finished devel/llvm22@default | llvm22-22.1.2: Success . . . [00:31:45] [release14-amd64-alt] [2026-03-27_22h11m23s] [committing] Time: 00:31:43 Queued: 90 Inspected: 0 Ignored: 0 Built: 90 Failed: 0 Skipped: 0 Fetched: 0 Remaining: 0 . . . 14.3-RELEASE official pkgbase jail-world installation: (a manual linewrap) [00:00:01] Inspecting /usr/local/poudriere/data/.m/release14-amd64-alt/ref//usr/ports for modifications to git checkout... no [00:00:02] Ports top-level git hash: 942c4a21a22ead9e98c150a9388b2eba71a75af2 . . . [00:05:16] [01] [00:00:00] Building devel/llvm22@default | llvm22-22.1.2 [00:26:01] [01] [00:20:45] Finished devel/llvm22@default | llvm22-22.1.2: Success . . . [00:26:05] [release14p3-amd64-alt] [2026-03-27_22h59m58s] [committing] Time: 00:26:03 Queued: 90 Inspected: 0 Ignored: 0 Built: 90 Failed: 0 Skipped: 0 Fetched: 0 Remaining: 0 . . . So: no where near the time-ratio that you got for your llvm19 experiment: 31min 45sec / 26min 5sec . Personal non-debug boot kernel build (main) is of: (some manual line wrapping) # uname -apKU FreeBSD 7950X3D-ZFS 16.0-CURRENT FreeBSD 16.0-CURRENT #2 main-n284691-f6989880841b-dirty: Wed Mar 25 13:34:30 PDT 2026 root@7950X3D-ZFS:/usr/obj/BUILDs/main-ZNV4-nodbg-clang/usr/main-src/ amd64.amd64/sys/GENERIC-NODBG amd64 amd64 1600014 1600014 # ~/fbsd-based-on-what-commit.sh -C /usr/main-src/ f6989880841b (HEAD -> main, freebsd/main, freebsd/HEAD) tests/netinet: add raw:reconnect test Author: Gleb Smirnoff <glebius@FreeBSD.org> Commit: Gleb Smirnoff <glebius@FreeBSD.org> CommitDate: 2026-03-25 19:58:28 +0000 branch: main merge-base: f6989880841b89d54ebcd5d12195c40a23627646 merge-base: CommitDate: 2026-03-25 19:58:28 +0000 n284691 (--first-parent --count for merge-base) Boot world (main) is from an official pkgbase world distribution, not a personal build (so it is a debug main world): # ~/pkgbase-snapshot-list.sh . . . RECENT INSTALLED: Via pkg-static info -C -x '^FreeBSD-' ( but no FreeBSD-set- ) . . . 9 FreeBSD-*-16.snap20260324220950 . . . As for the ports tree used: # ~/fbsd-based-on-what-commit.sh -C /usr/ports-alt 942c4a21a22e (HEAD -> main, freebsd/main, freebsd/HEAD) databases/pg_textsearch: Update to 1.0.0 Author: Kevin Bowling <kbowling@FreeBSD.org> Commit: Kevin Bowling <kbowling@FreeBSD.org> CommitDate: 2026-03-28 04:54:12 +0000 branch: main merge-base: 942c4a21a22ead9e98c150a9388b2eba71a75af2 merge-base: CommitDate: 2026-03-28 04:54:12 +0000 n739402 (--first-parent --count for merge-base)
side issue: git.madhouse-project.org is not responding on port 443
(In reply to Mark Millard from comment #53) Some adjustments: Should have been plural: repositories initially empty. The time ratio to match yours possibly should have not been across all 90 packages built but just for lang/llvm22: 25min 16sec / 20 min 45sec In: QUOTE So: no where near the time-ratio that you got for your llvm19 experiment: 31min 45sec / 26min 5sec . END QUOTE I listed only one of my time ratios but it reads like I was listing yours: 1h58min / 20min [Some automatic line wrapping happened that I did not intend, at least for how it displays here.]
(In reply to Mark Millard from comment #55) Also: I typed lang/llvm prefixes 5 times when they should have been devel/llvm prefixes. They were not lang/rust references.
(In reply to Sebastian Oswald from comment #51) Given the large difference in results for my testing that was based on officially distributed world builds vs. your report, I'm wondering if you have ever observed the unusually large build times with only officially-distributed world builds involved (not personally built worlds). Although my personal kernel/world builds are based on META_MODE (possibly with a prior rm -fr of the build tree), I do not use ccache or the like for building port-packages. Only the 3 "null" method poudriere-devel jails below are based on personal world builds. All 3 are based on main. The rest of the poudiere-devel jails are from using official world distributions. # poudriere jail -l you have mail JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP PATH release14p3-amd64 14.3-RELEASE-p10 1403000 amd64 ftp-archive 2026-03-27 15:05:00 /usr/local/poudriere/jails/release14p3-amd64 official14-amd64 14.3-STABLE 1404500 amd64 freebsdci 2026-03-24 23:52:30 /usr/local/poudriere/jails/official14-amd64 official14-i386 14.3-STABLE 1404500 i386 freebsdci 2026-03-24 23:43:57 /usr/local/poudriere/jails/official14-i386 release14-amd64 14.4-RELEASE 1404000 amd64 ftp-archive 2026-03-24 23:55:36 /usr/local/poudriere/jails/release14-amd64 release-amd64 15.0-RELEASE-p4 1500068 amd64 pkgbase 2026-03-24 23:56:32 /usr/local/poudriere/jails/release-amd64 official-amd64 15.0-STABLE 1500506 amd64 pkgbase 2026-03-24 23:12:17 /usr/local/poudriere/jails/official-amd64 main-ZNV4 16.0-CURRENT 1600014 amd64 null 2025-02-12 16:03:46 /usr/obj/DESTDIRs/main-ZNV4-poud main-ZNV4-bulk_a 16.0-CURRENT 1600014 amd64 null 2025-02-12 16:03:46 /usr/obj/DESTDIRs/main-ZNV4-poud-bulk_a main-ZNV4-dbg 16.0-CURRENT 1600014 amd64 null 2025-04-02 09:20:08 /usr/obj/DESTDIRs/main-ZNV4-poud-dbg main-amd64 16.0-CURRENT 1600014 amd64 pkgbase 2026-03-24 20:40:16 /usr/local/poudriere/jails/main-amd64 main-min-devel-lib32-amd64 16.0-CURRENT 1600014 amd64 pkgbase 2026-03-24 23:45:28 /usr/local/poudriere/jails/main-min-devel-lib32-amd64 I also have /usr/obj/DESTDIRs/main-ZNV4-chroot-ports-local/ that is based on a personally built world. But what I boot is an officially distributed main build, even though it is a debug build of main's world. (I sometimes use /etc/malloc.conf to cause junk:false for jemalloc for that context.) pkgbase for main supplies both debug and non-debug kernels and I normally use the non-debug one if I'm not booted with a personally-built kernel. For personally built kernels, I normally boot the non-debug variant when I use such a kernel. (If official pkgbase distributions ever started also distributing a non-debug build of main's world, I'd use it.)
I did test-builds on the 18th of March and it built (except 13.5, which we'll ignore for the time being). So, any showstoppers or can this go in ?
(In reply to Kurt Jaeger from comment #58) > So, any showstoppers or can this go in ? Thanks for testing. Yes, it can go, but this version is outdated, current latest release is 3.3.0. version 3.1.0 is from 3 months ago.
(In reply to Yusuf Yaman from comment #59) So, should we wait for 3.3.0 or ... ?
Created attachment 269292 [details] www/iocaine: add new port (version 3.1.0) I just had a look at the 3.2.0 and 3.3.0 versions. Version 3.3.0 has some major refactoring changes: https://git.madhouse-project.org/iocaine/iocaine/releases/tag/iocaine-3.3.0 This requires some changes to the Makefile (from a quick glimps at least the post/pre-extract,build,install phases and the pkg plist as well as the patch for the default config). Version 3.2.0 builds and installs with minor changes (i.e. udate PORTVERSION and regenerate distinfo and Makefile.crates). But the resulting binary won't run and seems to contain some broken artifacts/paths from the build environment: ---- $ iocaine start thread 'main' (155770) panicked at /wrkdirs/usr/ports/www/iocaine/work/iocaine-3.2.0/cargo-crates/roto-0.9.0/src/codegen/mod.rs:593:43: called `Result::unwrap()` on an `Err` value: Backend(unable to make memory readable+executable Caused by: System call failed: Permission denied (os error 13)) note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace Abort (core dumped) ---- I haven't looked further into this, but I'll give the update to 3.3.0 a shot over the easter weekend. Since branching of 2026Q2 is imminent, I suggest we get the new port committed with version 3.1.0 and then tackle the upgrade to 3.3.0 afterwards. If this is still in time for 2026Q2, thats a bonus; if not, we at least have www/iocaine in quarterly. I attached a slighly updated patch for 3.1.0 with the following improvements: - containing the 'diff for some adjustments' by @nxjoseph - bumped ROBOTS_VERSION to 1.45 - added ai-robots-txt-path to (example) config.kdl - fixed config file extension in rc file
(In reply to Sebastian Oswald from comment #61) This fails with the checksums in distinfo ? ===> Giving up on fetching files: robots.json
Created attachment 269325 [details] version 3.3.0 I've tried to update it to latest stable release v3.3.0. I've done some refactoring to Makefile for setting UP cratesdir and WRKSRC. I have now run `iocaine start` and as my user "yusuf", /var/db/iocaine (owned by yusuf). it seems to start fine and it listens on port 42069. I'd appreciate some testing and runtime test, thank you. also a note: ${WRKSRC}/data/defaults/etc -> ${WRKSRC}/iocaine-powder/embeds/defaults/etc I guess this is the new right directory to use? I also would appreciate a to-do list on how to test the port runtime, if you don't have time and/or busy etc. This patch is tested on Poudriere (14.3-RELEASE-p9, amd64, main (c927d063a7ee)) and it seems OK. portlint, portclippy, portfmt are satisfied. Thanks.
(In reply to Kurt Jaeger from comment #48) FYI: 135i386-default (a.k.a. latest) is now a failure case for building curl on the official builders. For example e46d80fd517c (still in process) and the earlier 89a804c19f17 and ea4da10cbbca : --- libcurlu_la-md5.lo --- In file included from md5.c:76: In file included from /usr/local/include/wolfssl/openssl/md5.h:32: In file included from /usr/local/include/wolfssl/wolfcrypt/hash.h:29: /usr/local/include/wolfssl/wolfcrypt/types.h:1636:6: error: "bad math long / long long settings" 1636 | #error "bad math long / long long settings" | ^ /usr/local/include/wolfssl/wolfcrypt/types.h:1638:1: error: use of empty enum 1638 | }; | ^ 2 errors generated. *** [libcurlu_la-md5.lo] Error code 1 The rest of the 135*-default and 135*-quarterly built. So curl failures should now be limited to only i386 13.5 for official builds. (The new 2026Q2 for 13.5 likely will have the same i386 issue, at least for now.)
(In reply to Yusuf Yaman from comment #63) did you testbuild in poudriere ? it fails with distinfo out of date ?
(In reply to Kurt Jaeger from comment #65) yes i did test the patch in poudriere, also just now, ran `make distclean fetch` and it completed fine.
(In reply to Yusuf Yaman from comment #63) Hi, your latest patched 3.3.0 version builds fine on [00:00:00] Poudriere version: poudriere-git-3.4.99.20260304 [00:00:00] Host OSVERSION: 1500506 [00:00:00] Jail OSVERSION: 1500504 hope to test it on at least one web server in the next couple of days
(In reply to Kurt Jaeger from comment #62) It seems since the filename (robots.yaml) doesn't have any versioning information, it won't get re-downloaded if already present in the distfiles. A 'make distclean' solves it for manual builds; it seems poudriere builds start with a clean environment anyways. I'll give the 3.3.0 version a shot tomorrow on one of the webservers that's already running 3.1.0.