After upgrading the Jellyfin 10.10.1 on freebsd-14.1/amd64 i've been unable to scan my media library. this worked on prior versions, so suspect something changed with this latest update. the symptoms are: 1. select scan library using webUI 2. after 30seconds or so the application crashes and is restarted 3. "Nov 12 01:23:35 HOSTNAME kernel: pid 92570 (jellyfin), jid 0, uid 868: exited on signal 11 (no core dump - other error)" is printed in /var/log/messages 4. no new media detected i've tried this on a completely fresh install and when i attempt to scan media which was working fine prior to the upgrade i get the same behavior. i can enable debug logging if needed, but it's pretty verbose so wanted to log a bug here first.
Downgrading to the previous version "10.9.11" works, so it seems like there is a regression in 10.10. I've been able to rescan my entire library that is several TB's and haven't had any crashes.
I cannot reproduce, I would need more info to be able to reproduce, I am still fresh to dotnet/jellyfin world, I don't know what to ask for, but maybe look at jellyfin logs.
(In reply to Baptiste Daroussin from comment #2) I did a bit more digging, I can reproduce this on a fresh VM. I start with a clean install (remove /var/db/jellyfin) - go through the setup wizard. when i add a video media library it seems to segfault when extracting metadata from the file. running jellyfin directly from a shell exits like this: [21:07:10] [INF] [21] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Starting ffprobe with args -analyzeduration 200M -probesize 1G -i file:"/tmp/movies/The.Thing.1982.REMASTERED.1080p.BluRay.6CH.ShAaNiG.mkv" -threads 0 -v warning -print_format json -show_streams -show_chapters -show_format Segmentation fault (core dumped) $ interestingly enough i run the above "ffprobe" command in the same shell and it outputs file in JSON as expected, so don't think that's the issue. i loaded the core into gdb and got some info (but no debug symbols so that's probably the next step): Core was generated by `/usr/local/jellyfin/jellyfin -d /var/db/jellyfin/ -C /var/cache/jellyfin/'. Program terminated with signal SIGSEGV, Segmentation fault. Address not mapped to object. #0 0x0000004922b195f0 in jpeg_core_output_dimensions () from /usr/local/lib/libjpeg.so.8 [Current thread is 1 (LWP 102854)] (gdb) bt #0 0x0000004922b195f0 in jpeg_core_output_dimensions () from /usr/local/lib/libjpeg.so.8 #1 0x0000004922b19b59 in jpeg_calc_output_dimensions () from /usr/local/lib/libjpeg.so.8 #2 0x0000004922895614 in ?? () from /usr/local/lib/libSkiaSharp.so #3 0x00000049224dda60 in sk_codec_get_scaled_dimensions () from /usr/local/lib/libSkiaSharp.so #4 0x000000080d6aaf59 in ?? () #5 0x00000000021321f9 in ?? () #6 0x0000000802651b68 in ?? () from /usr/local/jellyfin/libcoreclr.so #7 0x00007fffdd1a4d90 in ?? () #8 0x00000008074f9f28 in ?? () #9 0x0000004923c85ea8 in ?? () #10 0x00007fffdd1a27b0 in ?? () #11 0x000000080d6aaf59 in ?? () #12 0x00007fffdd1a2850 in ?? () #13 0x0000000886c45298 in ?? () #14 0x000000491e2e1800 in ?? () #15 0x000000491de04c00 in ?? () #16 0x0000000000000080 in ?? () #17 0x0000000000000000 in ?? () (gdb) my wild guess is something funky is happening with libskiasharp after being broken out into its own package? Running this through truss and it seems like the segfault is happening when it tries to create some thumbnail images: open("/var/db/jellyfin/metadata/People/K/Kurt Russell/folder.jpg",O_RDONLY,0666) = 535 (0x217) lseek(535,0x0,SEEK_CUR) = 0 (0x0) fstat(535,{ mode=-rw-r--r-- ,inode=1763525,size=126669,blksize=32768 }) = 0 (0x0) fstat(535,{ mode=-rw-r--r-- ,inode=1763525,size=126669,blksize=32768 }) = 0 (0x0) lseek(535,0x0,SEEK_CUR) = 0 (0x0) lseek(535,0x18000,SEEK_SET) = 98304 (0x18000) read(535,"SNIP"...,32768) = 28365 (0x6ecd) lseek(535,0x0,SEEK_SET) = 0 (0x0) pread(535,"SNIP",32,0x0) = 32 (0x20) sigprocmask(SIG_BLOCK,0x0,{ }) = 0 (0x0) pread(535,"SNIP...,1024,0x0) = 1024 (0x400) mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_SHARED,9,0x4552000) = 34382782464 (0x8015fa000) munmap(0x8015b3000,4096) = 0 (0x0) mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_SHARED,9,0x454a000) = 34382491648 (0x8015b3000) munmap(0x4921fb0000,8192) = 0 (0x0) SIGNAL 11 (SIGSEGV) code=SEGV_MAPERR trapno=12 addr=0x14 Happy to provide a VM image you can run in bhyve if that'll help debug this, otherwise i guess i'll try to figure out how to compile jellyfin and dependencies with debug symbols tomorrow. thanks!
No your backtrace is enough, I can investigate now!
are you building locally or are you using official packages ?
(In reply to Baptiste Daroussin from comment #5) I’m using the official “latest” packages on 14 amd64.
(In reply to Baptiste Daroussin from comment #5) FWIW, I have the same issue with 10.10.1 and libskiasharp both built locally
Hi Baptiste It was me who asked you add 10.10.x to the 13.x ports, last week. And I also have had this problem, with not beeing able to scan/detect new files. Adding a new test-lib resoulted in "permission errors", but also created new folders. Every time I hit okay and it failed the system created a new folder, adding 2,3,4,5,6 I rolled back my jail and things are fine again since then. I only reported this over here (was unsure about who is contrib. to JF): https://github.com/Thefrank/jellyfin-server-freebsd/issues/89#issue-2656126899
I just finished debugging and writing this up. https://github.com/jellyfin/jellyfin/issues/13016#issuecomment-2480893595 https://github.com/libjpeg-turbo/libjpeg-turbo/issues/795 tl;dr: jellyfin -> libskiasharp -> skia calls into libjpeg with a "fake" partially-initialised decompress struct. mozjpeg is fine, but jpeg-turbo dereferences a NULL field and crashes. I believe this to be a jpeg-turbo bug, and have report it. Workaround for FreeBSD for now might just to be to relink libskiasharp against mozjpeg (I guess?). Doesn't help other Skia users, I suppose. (I don't really know anything, just had a few spare hours and was annoyed that my server kept crashing).
Dug a bit more today. As best I can tell, when you tell Skia to build with its own jpeg-turbo and not the system one, it will use a repo pinned at 2.1.0. https://skia.googlesource.com/external/github.com/libjpeg-turbo/libjpeg-turbo/ In that version, the NULL dereference doesn't happen: https://github.com/libjpeg-turbo/libjpeg-turbo/blob/2.1.x/jdmaster.c#L101 So I suppose the answer is that libskiasharp should build with: skia_use_system_libjpeg_turbo=false However, just changing this in ports/graphics/libskiasharp/Makefile and rebuilding yields: ninja: error: '../third_party/externals/libjpeg-turbo/jaricom.c', needed by 'obj/third_party/externals/libjpeg-turbo/libjpeg.jaricom.o', missing and no known rule to make it So I guess that bit needs to be pulled separately.
Today's updates! jpeg-turbo maintainer believes that Skia is using the API wrong: https://github.com/libjpeg-turbo/libjpeg-turbo/issues/795#issuecomment-2484148592 I've logged a bug against Skia: https://issues.skia.org/issues/379669745 I assume from all this that there's no way Skia will be working properly against jpeg-turbo 3.x in the near future, so I expect the only option for Jellyfin for now is to stick to 2.x (by whatever means).
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=58625b6da7a6c3b4c1fea0e7fb0bd97be77bb24c commit 58625b6da7a6c3b4c1fea0e7fb0bd97be77bb24c Author: Baptiste Daroussin <bapt@FreeBSD.org> AuthorDate: 2024-11-19 09:24:01 +0000 Commit: Baptiste Daroussin <bapt@FreeBSD.org> CommitDate: 2024-11-19 09:35:27 +0000 graphics/libskiasharp: bundle chromium's libjpeg-turbo version Apparently libskasharp (abuses the libjpeg-turbo's API) which results in some crashes: - https://github.com/Thefrank/jellyfin-server-freebsd/issues/89 - https://github.com/libjpeg-turbo/libjpeg-turbo/issues/795 - https://github.com/jellyfin/jellyfin/issues/13016 PR: 282704 graphics/libskiasharp/Makefile | 13 ++++++++++++- graphics/libskiasharp/distinfo | 4 +++- 2 files changed, 15 insertions(+), 2 deletions(-)
thank you all for the help on the diagnostic, please tell me if the updated version fixes the issue for you ?
Thanks Baptiste. A ran a full rescan overnight, and everything worked as I'd expect. Great work, thanks a lot!
Thanks everyone who helped with this issue, and huge thanks to bapt@ for fixing it up. I can confirm the upgrade works on my local VM i was testing this on. Closing the bug!