Bug 232966

Summary: multimedia/emby-server: Bundle x264
Product: Ports & Packages Reporter: Maxime Gauduin <alucryd>
Component: Individual Port(s)Assignee: Li-Wen Hsu <lwhsu>
Status: Closed FIXED    
Severity: Affects Many People CC: alucryd, driesm.michiels, joshruehlig, lwhsu, ygy
Priority: --- Keywords: needs-qa
Version: LatestFlags: lwhsu: maintainer-feedback+
Hardware: amd64   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232935
Bug Depends on:    
Bug Blocks: 232935    
Description Flags
revision 2 diff
Work in progress diff
WIP v2
none diff
Using port variables not overwriting do-configure
upgrade emby-server to latest and greatest package
update to latest and greatest
update to latest and greatest none

Description Maxime Gauduin 2018-11-04 19:03:04 UTC
Created attachment 198947 [details]
revision 2 diff

The current package was built against quarterly, but our ffmpeg build fails to load on latest because of the last x264 update (see bug #232935). To better follow our release cycle we decided it would be better to ship our own x264 library as it is undergoing a soname bump with most releases. We still depend on system libs for the rest.

We also removed the message about imagemagick and ffmpeg, as ffmpeg is now included and imagemagick has been replaced with skiasharp (also included).
Comment 1 Bugzilla Automation freebsd_committer 2018-11-04 19:03:04 UTC
Maintainer informed via mail
Comment 2 joshruehlig 2018-11-09 18:14:05 UTC
Hey Maxime,
Thanks for helping with this port. I have been the one updating (old PBI version of) FreeNAS plugin for emby. I have been attempting to update the PBI version of the plugin to and in the process realized that the way things are now structured leads itself to dependency issues.

Is it not possible to just include all necessary libraries for ffmpeg to execute? I think this is how plex is distributed, including it's own ffmpeg. Otherwise we will constantly be hitting issues where installed dependencies are not the exact version expected.

I do not know if this suggestions would violate certain dependencies' licenses. In that case, maybe we could instead specify the exact library version needed.
But, this still isn't perfect because if opus was updated, we would need to update the emby-server port as well.

Comment 3 joshruehlig 2018-11-09 18:18:32 UTC
Also, the current emby exe will not execute with just any mono. We should specify the minimum version in the port. I believe the below should work.
Comment 4 Dries Michiels 2018-11-10 10:42:07 UTC
I agree, as long as we have shared library dependencies we will observe the same issues coming back. Ideally, the emby-server package bundles everything if licences permit.
Comment 5 Ben Woods freebsd_committer 2018-11-10 11:27:32 UTC
Waiting to hear back from Maxime where any further changes are going to be proposed with the diff before committing (as per conversation with Josh and Dries above).
Comment 6 Ben Woods freebsd_committer 2018-11-10 11:29:15 UTC
Hi Maxime,
The other change I would recommend is that the MAINTAINER email address be set to the one you use here on Bugzilla, so any bug reports will notify you automatically.
Should the port be changed so that MAINTAINER=alucryd@archlinux.org or will you open a bugzilla account with username apps@emby.media?
Comment 7 Maxime Gauduin 2018-11-11 10:27:06 UTC
Hi all,

I have in fact another proposal, I will push this custom ffmpeg to Arch Linux in the future, and absolutely need to build it from source, so the source code will not just be available upon request but publicly available.

I would like to build this custom ffmpeg in the FreeBSD port too, this way it will always be rebuilt against the binary system libraries and we won't face this issue anymore. I really don't want to duplicate the effort you've all spent building and patching the libraries we need by bundling all of them.

@Ben Thanks for the feedback, I will change the maintainer email address to mine with the next update.
Comment 8 Dries Michiels 2018-11-14 13:38:03 UTC
I love where we are headed to fix it permanently in the future.
I understand that this isn't an easy task to do in the short term.
Maybe we can commit the patch as is to fix it temporarily?
Comment 9 Guangyuan Yang freebsd_committer 2018-11-28 08:09:55 UTC
Also hit by this problem and would love to see a permanent fix.

IANAPC (I am not a ports committer), but if I understand it correctly, if at least a temporary fix is not committed by the end of the year, it could also break latest, right?
Comment 10 Maxime Gauduin 2018-11-28 19:51:26 UTC
Yes it will.

I will try to accelerate things on our end, I still need a public tarball of our ffmpeg to be produced, I will then proceed to upgrading the port with it.

Our 3.6 release is just around the corner, I would like to get this done by then.
Comment 11 Maxime Gauduin 2018-12-14 11:40:56 UTC
Good news, I finally have a public ffmpeg tarball to play with.

Quick question guys, if I were to build our custom ffmpeg inside the emby port, would I be able to build against libmp3lame even if it's not in the binary repositories? If not I'll need to bring libmp3lame along with it's headers and pkgconfig files which I really want to avoid.
Comment 12 Dries Michiels 2018-12-14 11:45:34 UTC
You can find libmp3lame in the audio/lame port.
If you define a run_dependency against that it will get linked.

[/usr/ports/audio/lame]$ cat pkg-plist | grep libmp3lame
Comment 13 Maxime Gauduin 2018-12-14 19:37:50 UTC
Created attachment 200124 [details]
Work in progress diff

Alright, I've been working on building ffmpeg directly. This is not the final diff, just to get some opinions. It still uses the tarball which contained ffmpeg and its libs along with libx264 and libmp3lame. They will be gone with the next release which will happen soon. It will only contain the partially static ffmpeg built in the Makefile.

One worthy note, we will be reverting to using imagemagick until skia with dng support can be built on freebsd. I'm still pondering whether to use the system one (which is not built with ideal options) or ship our own.
Comment 14 Dries Michiels 2018-12-16 09:07:59 UTC
Hi, I think the general approach is going the right way! I tried building the port in poudriere but it gave some errors. Also portlint -ac gives a few minor warnings. We use poudriere and portlint to check for QA. Could you check these out and update the diff accordingly.

This is the output of portlint -ac:
root@freebsd-head:/usr/ports/multimedia/emby-server # portlint -ac
WARN: /usr/ports/multimedia/emby-server/pkg-plist: [3]: installing shared libraries, please define USE_LDCONFIG as appropriate
WARN: Makefile: [94]: possible direct use of command "ln" found. use ${LN} instead.
WARN: Makefile: using hyphen in PORTNAME. consider using PKGNAMEPREFIX and/or PKGNAMESUFFIX.
WARN: /usr/ports/multimedia/emby-server/pkg-deinstall: possible use of absolute pathname "/var/db/emby-server".

Poudriere fails with this (don't know how to fix this, Ben could maybe give some feedback about this part):
=======================<phase: build          >============================
===>  Building for emby-server-
(cd /wrkdirs/usr/ports/multimedia/emby-server/work/ffmpeg-2018_12_01 &&  ./configure  --disable-bzlib  --disable-doc  --disable-ffplay  --disable-shared  --disable-vdpau  --disable-libxcb  --disable-libxcb-shm  --disable-libxcb-xfixes  --disable-libxcb-shape  --disable-lzma  --disable-sdl2  --disable-xlib  --enable-fontconfig  --enable-gmp  --enable-gnutls
--enable-gpl  --enable-libass  --enable-libdrm  --enable-libfreetype  --enable-libfribidi  --enable-libmp3lame  --enable-libopus  --enable-libsmbclient  --enable-libtheora  --enable-libvorbis  --enable-libwebp  --enable-libx264  --enable-libzvbi  --enable-static  --enable-vaapi  --enable-version3 &&  gmake)
gcc is unable to create an executable file.
If gcc is a cross-compiler, use the --enable-cross-compile option.
Only do this if you know what cross compiling means.
C compiler test failed.
Comment 15 Dries Michiels 2018-12-16 09:10:04 UTC
WARN: Makefile: using hyphen in PORTNAME. consider using PKGNAMEPREFIX and/or PKGNAMESUFFIX. This one should not be fixed I think ;)
Comment 16 Maxime Gauduin 2018-12-18 13:12:17 UTC
Created attachment 200228 [details]
WIP v2

Added USE_LDCONFIG and replaced ln with ${LN}. The absolute path in pkg-deinstall is just a comment, nothing to worry about.

I made some changes and tried poudriere, but it fails to build llvm60 here, and takes 4 hours to do so, so I gave up. Could you give the latest diff a try?
Comment 17 Dries Michiels 2018-12-25 09:58:06 UTC
When is the new release coming out? We should try and push the bundled ffmpeg with the new version in the tree. I reached out to the multimedia mailing list to help us push this in the tree.
Comment 18 Guangyuan Yang freebsd_committer 2018-12-28 15:14:58 UTC
Heads up - not much time left before breaking the new quarterly :)
Comment 19 Maxime Gauduin 2019-01-22 17:25:51 UTC
Created attachment 201338 [details] diff

Hi folks, we finally got around publishing a new release, this latest diff should be the final one. Unfortunately I still can't test poudriere as it keeps failing to build llvm, so it would be awesome if someone else could. Portlint at least seems to be happy, and I can build and install the port fine locally.
Comment 20 Dries Michiels 2019-01-22 19:09:54 UTC
On what version of FreeBSD are you testing this? Also what ports tree have you checked out? Head or quarterly? Thanks
Comment 21 Maxime Gauduin 2019-01-23 19:05:07 UTC
I'm testing on Freebsd11.2 latest cycle, with the head of the ports repository:

uname -a
FreeBSD freebsd11 11.2-RELEASE FreeBSD 11.2-RELEASE #0 r335510: Fri Jun 22 04:32:14 UTC 2018     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

svn info /usr/portsPath: .
Working Copy Root Path: /usr/ports
URL: https://svn.freebsd.org/ports/head
Relative URL: ^/head
Repository Root: https://svn.freebsd.org/ports
Repository UUID: 35697150-7ecd-e111-bb59-0022644237b5
Revision: 487425
Node Kind: directory
Schedule: normal
Last Changed Author: joneum
Last Changed Rev: 487425
Last Changed Date: 2018-12-14 11:57:19 +0000 (Fri, 14 Dec 2018)
Comment 22 Dries Michiels 2019-01-28 21:18:53 UTC
Created attachment 201491 [details]
Using port variables not overwriting do-configure

Still some problems with shared libs, can't get samba to work error out with undefined symbols in its shared library. WIP
Comment 23 Dries Michiels 2019-02-06 20:41:40 UTC
Created attachment 201800 [details]
upgrade emby-server to latest and greatest package

Attached patch uses port variables and adds 2 options (one for lame defaulting to off and one for samba defaulting to on). Lame defaults to off for licensing and samba defaults to on for full functionality of emby-server. Pkg-message adjusted to recommend compiling from ports with lame enabled for full functionality.

Portlint -ac
poudriere testport head
poudriere testport stable12
poudriere testport stable11

Ready to commit. Now only a developer is needed to commit it and give his approval on the suggested approach.
Comment 24 Dries Michiels 2019-02-07 20:03:52 UTC
Created attachment 201822 [details]
update to latest and greatest

Forgot to add the sub files to the diff.
Comment 25 Dries Michiels 2019-02-08 19:46:01 UTC
Created attachment 201850 [details]
update to latest and greatest

Add X11 option as this is also a much used option in /etc/make.conf
Comment 26 Li-Wen Hsu freebsd_committer 2019-02-12 02:46:36 UTC
(In reply to Dries Michiels from comment #25)

https://bugs.freebsd.org/bugzilla/attachment.cgi?id=201850&action=diff tested fine in my poudriere and ygy@ also verified it works.  The only thing I want to check is, this patch removes "Created by" comment, is that ok'd by Ben?
Comment 27 Dries Michiels 2019-02-12 07:54:59 UTC

Please use the diff from the review, mat went over some of the things. Much improved version :-). https://reviews.freebsd.org/D19130

As for the consent of Ben, he gave this in the following PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231669

Although, I don't actually mind that the line stay as well as the maintainer.
Comment 28 Dries Michiels 2019-02-25 20:45:37 UTC
Hi guys, I'd like to give a friendly bump on this PR. 

Li-Wen Hsu, would you mind committing the patch that is in the phabricator review? I don't think there will be any more comments on the diff.

Comment 29 Dries Michiels 2019-03-04 12:31:17 UTC
If mat@ has no further comment on the review I think this can be committed.
And share the love of an updated emby-server with everyone ;).

Thanks guys!
Comment 30 Li-Wen Hsu freebsd_committer 2019-03-04 14:08:03 UTC
(In reply to Dries Michiels from comment #29)
I also think it's fine to get in the tree.  After reading the comments in this one and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232935 , the patch in https://reviews.freebsd.org/D19130 fixes both issues, do I understand correctly?

Also, Maxime, could you do the final maintainer approval for the patch in https://reviews.freebsd.org/D19130 ?
Comment 31 Maxime Gauduin 2019-03-04 15:07:21 UTC
(In reply to Li-Wen Hsu from comment #30)
Registered and approved the diff over there, hopefully I got it right, this was a first for me.
Comment 32 Dries Michiels 2019-03-04 15:25:41 UTC
(In reply to Li-Wen Hsu from comment #30)
Exactly! This update fixes that issue as we are now building the ffmpeg locally and always link against the (updated) correct shared library version.
Comment 33 Li-Wen Hsu freebsd_committer 2019-03-04 15:28:50 UTC
Dries, thanks for the explanation.  I'm working on it right now.

Maxime, thanks for the approval, in fact I was asking an approval here, which is enough.  It's even better you open a phabricator account, I'm looking forward to seeing more patches from you. :-)
Comment 34 commit-hook freebsd_committer 2019-03-04 20:15:41 UTC
A commit references this bug:

Author: lwhsu
Date: Mon Mar  4 20:14:40 UTC 2019
New revision: 494643
URL: https://svnweb.freebsd.org/changeset/ports/494643

  - Update to [1]
  - Fix cannot fix shared library errors [2] [3]
  - Update maintainer's email [3]

  PR:		232935 [2] 232966 [3]
  Submitted by:	Dries Michiels <driesm.michiels@gmail.com> [1] [2]
  		Maxime Gauduin <alucryd@archlinux.org> [3]
  Approved by:	Maxime Gauduin <alucryd@archlinux.org> (maintainer)
  Differential Revision:	https://reviews.freebsd.org/D19130

Comment 35 Dries Michiels 2019-03-04 20:53:23 UTC
Thank you for committing this!
Comment 36 Guangyuan Yang freebsd_committer 2019-03-05 22:15:43 UTC
(In reply to Li-Wen Hsu from comment #33)

Thank you Li-Wen!!