Bug 211997 - Mk/bsd.tex.mk: Update to TeX Live 2019
Summary: Mk/bsd.tex.mk: Update to TeX Live 2019
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Hiroki Sato
URL: https://reviews.freebsd.org/D16623
Keywords: needs-patch, needs-qa
: 212633 (view as bug list)
Depends on: 220215 220692
Blocks:
  Show dependency treegraph
 
Reported: 2016-08-19 13:46 UTC by bsd
Modified: 2019-07-08 21:07 UTC (History)
22 users (show)

See Also:
bugzilla: maintainer-feedback? (hrs)


Attachments
Texlive 2017 update (bzipped) (261.26 KB, patch)
2017-07-21 14:19 UTC, Zsolt Udvari
no flags Details | Diff
Texlive 2018 gzipped update (01) (564.88 KB, application/gzip)
2018-08-08 10:41 UTC, Zsolt Udvari
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description bsd 2016-08-19 13:46:57 UTC
Would it be possible to update TeXLive-full to TL2016 (TeXLive 2016)?

https://www.tug.org/texlive/acquire.html

Thank you.
Comment 1 bsd 2016-10-14 14:40:15 UTC
Is there anything I can do to help?

I cannot code, but perhaps testing?
Comment 2 Alan Braslau 2016-11-12 01:23:03 UTC
TeXlive is updated once a year, every year, and if the freeBSD port is going to be very useful, it is important that it be updated promptly as well.

TeXlive can be a monster of a package, but we build (and test) it for FreeBSD so it should mostly work without much modification.

We all appreciate the maintainer(s) of this package including TeX in the ports tree. Thank you for your efforts.

Alan
Comment 3 O. Hartmann 2016-12-02 09:23:42 UTC
(In reply to Alan Braslau from comment #2)

I ran into slight problems using TeXlive 2015 from ports in cunjucntion with some templates we were provided with from our publisher a couple of manths ago. These templates are mostly provided for scientific texts or book chapters, also with columns. We helped ourselfs then using the Linux system in our bureau which had already TeXlibe 2016 installed.

I would appreciate a refurbishment of the port, not necessarily TL2016 at the end of the year 2016 and according to the next scheduled TL2017 distribution and having in mind the effords need to be taken to provide a clean port.
Comment 4 Alan Braslau 2016-12-02 18:22:10 UTC
(In reply to O. Hartmann from comment #3)

The templates provided by publishers are generally notoriously bad and may not work correctly in any case. However, in your case, it sounds like the templates that you use might require features present in TL2016. There may not be anything wrong (or "not clean") with the present port; it may simply need updating in order to follow developments in TeXLive.

Please note that the TeXLive tree is organized so that it be portable and can be installed anywhere on your system. It can co-exist with the present port without conflict: the instance that will be used depends only on your PATH and eventually on other environment variables. Although it would be cleaner for the port to be updated, you can always install the latest TL from tug.org.
Comment 6 Kurt Jaeger freebsd_committer 2017-06-11 20:22:34 UTC
See also:

http://anthesphoria.net/FreeBSD/TeXLive-2017/
Comment 7 Kurt Jaeger freebsd_committer 2017-06-11 20:23:37 UTC
*** Bug 212633 has been marked as a duplicate of this bug. ***
Comment 8 bsd 2017-07-01 15:26:43 UTC
Doing a manual installation, as shown in http://anthesphoria.net/FreeBSD/TeXLive-2017/ is fine, until one needs tools like Kile, TeXzilla, etc.

Those all depends tex-formats, on http://www.freshports.org/print/tex-formats/ which in turns force other 2015-old TeX binaries.

So, manual installation would not work by itself. One also needs to manually amend (like symlinking) the forced dependencies, since some of those tools look by exact path, not by local $PATH.

Manual symlinking would not leave a machine with a clean state.

It would be much better to have a proper texlive-full 2017 (out since June 4th).
Comment 9 Zsolt Udvari 2017-07-09 15:54:32 UTC
I'm working on a big texlive-update. It seems it will be right. Please be patient :)
Comment 10 O. Hartmann 2017-07-09 15:58:43 UTC
I'm pleased to read that and can't await the update ...
Comment 11 Zsolt Udvari 2017-07-21 14:19:05 UTC
Created attachment 184576 [details]
Texlive 2017 update (bzipped)

Update TeXLive* to 2017.
Some major changes:
* introduce print/texlive-base/bsd.texlive-helpers.mk
* every print/tex* and devel/tex* use bsd.texlive-helpers.mk so next time (I hope) the update will be easier
* fix licenses

Minor changes:
* fix pkg-plists
* fix versions
Comment 12 Zsolt Udvari 2017-07-21 14:26:28 UTC
(In reply to Zsolt Udvari from comment #11)
I could create packages, the poudriere testport are okay.
Portlint says some warning about slave port (PORTVERSION is defined in bsd.texlive-helpers.mk).

I can build my *.tex files to pdf (test on some not all) with pdflatex so I hope it works well.
I don't use many parts of TeXLive so I can't test them. If anybody has time to upgrade the needed packages please test it!

I couldn't (re)build the tlmgr database (check print/texlive-tlmgr/Makefile, ${TLPKG_FILE}) but don't know who needs when can use the FreeBSD's package manager. I think tlmgr should be used per-user not system-wide.
Comment 13 Zsolt Udvari 2017-07-21 14:31:02 UTC
(In reply to Zsolt Udvari from comment #11)
Sorry, I forgot the list of updated ports:
devel/tex-kpathsea
devel/tex-libtexlua
devel/tex-libtexluajit
devel/tex-synctex
devel/tex-web2c
print/tex-aleph
print/tex-basic-engines
print/tex-dvipdfmx
print/tex-dvipsk
print/tex-formats
print/tex-jadetex
print/tex-luatex
print/tex-ptexenc
print/tex-xdvik
print/tex-xetex
print/tex-xmltex
print/texlive-base
print/texlive-docs
print/texlive-full
print/texlive-texmf
print/texlive-texmf-source
print/texlive-tlmgr
Comment 14 Kurt Jaeger freebsd_committer 2017-07-23 04:15:22 UTC
While testbuilding, I got this:

Undefined subroutine &TeXLive::TLUtils::prepend_own_path called at /usr/local/bin/fmtutil line 66.
*** Error code 255

in print/tex-format.

I also had to fix print/tex-luatex/files/patch-Makefile.in
Comment 15 Zsolt Udvari 2017-07-23 10:21:19 UTC
(In reply to Kurt Jaeger from comment #14)
Hm, it works for me: http://freebsd.uzsolt.hu/logs/print/tex-formats/tex-formats-20170524.log
Do you test on FreeBSD 11?
Comment 16 Kurt Jaeger freebsd_committer 2017-07-23 10:51:08 UTC
I build on current. I'll investigate further.
Comment 17 Zsolt Udvari 2017-07-23 17:58:00 UTC
(In reply to Kurt Jaeger from comment #16)
Build on FreeBSD 11 too.
Comment 18 Hiroki Sato freebsd_committer 2017-07-24 03:52:35 UTC
I am sorry for not updating TeXLive-related ports in a timely manner.  I just wanted to let you know that I have resumed the updating work and reviewing the submitted patch now.
Comment 19 Kurt Jaeger freebsd_committer 2017-07-24 05:32:44 UTC
Build on 11.0:

[04:27:49] ====>> Failed ports: print/tex-luatex:patch print/tex-aleph:patch
[04:27:49] ====>> Skipped ports: print/tex-dvipdfmx print/tex-jadetex print/texlive-full
[04:27:49] ====>> Ignored ports: print/texlive-texmf-source print/tex-xetex print/tex-xmltex
Comment 20 Zsolt Udvari 2017-07-25 12:50:24 UTC
(In reply to Kurt Jaeger from comment #19)
I don't understand. For me it works - the patch and the build too.
Maybe did I create my patch wrong?
Comment 21 Zsolt Udvari 2017-08-08 07:53:33 UTC
(In reply to Hiroki Sato from comment #18)
How about with the patch?
Comment 22 w.schwarzenfeld freebsd_triage 2018-03-10 01:08:11 UTC
ping!
Comment 23 w.schwarzenfeld freebsd_triage 2018-03-10 01:09:26 UTC
See also bug #226023.
Comment 24 Zsolt Udvari 2018-03-11 08:39:10 UTC
(In reply to w.schwarzenfeld from comment #22)

My patch is somehow wrong (I promise it worked for me last summer).
I want to update TeXLive to 2017, I'm working on it but I don't have many (enough) free time.

I would have some idea (maybe it's worth to discuss them):

- would nice if the current version(s) will stored in Mk/bsd.tex.mk (or in any centralised file). Maybe two variables: for example TEXBINVERSION and TEXMFVERSION (they can differ), and more two variables: TEXBINDISTFILE and TEXMFDISTFILE. So doesn't need update the many-many tex*-releated ports if the 2018 will release (maybe soon).

- IMHO unnecessary split the bin package into many devel/tex-* and print/tex-* packages. I think the main part (which should include devel/tex-* and some of print/tex-*) can be ONE package: would be easier to maintain only one package (and its tons of patches). About every tex-* depends on kpathsea, web2c, ptexenc. I think nobody wants install only ptexenc (without texlive).
These splits requires many-many plus patch.

- the 1.7Gb texlive-texmf should split smaller (functional) ones. Its build process is only "copy the files to STAGEDIR" so it's easy to handle it. Maybe it's possible with FLAVORS. I'm using TeXLive 2017 (the packages is created some month ago with attached patch) and I think I don't use more than half.
The texlive-texmf-{doc,source} should split similar way as texmf-tree (or include the doc into releated texlive-texmf-* package). For example as ArchLinux (https://www.archlinux.org/packages/?q=texlive) or Debian (https://packages.debian.org/search?keywords=texlive&searchon=names&suite=stable&section=all).
The texlive-texmf is required by some packages to build their documentation. I think they needed only some files from the big 1.7Gb (more time to test/build).
Comment 25 w.schwarzenfeld freebsd_triage 2018-03-11 09:11:23 UTC
Thanks for answer!
Comment 26 Michael Osipov 2018-03-11 09:25:30 UTC
I came across this one and noticed that the or one of the maintainers always tests on FreeBSD too and writes why installing TL directly is better than through ports. One thing to note is that there are so many package options during the TL install which this port cannot cover at all. Personally, I am not willing to install 1,7 GB just because minimal does not suite, but another config with 300 MB happily suits my needs.
Comment 27 Kurt Jaeger freebsd_committer 2018-03-11 20:21:18 UTC
Link to the TL maintainer blogpost on TL 2017 and FreeBSD:

http://anthesphoria.net/FreeBSD/TeXLive-2017/
Comment 28 Zsolt Udvari 2018-05-02 15:20:30 UTC
TeXLive 2018 is released some days ago.
Maybe would be nice to edit the bug report's title (again).

I've some ideas (described in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211997#c24) and if there isn't other man (or woman :) ) and my opinions (#24) are logical I would do willingly the update - but only in June (in year 2018 of course).

https://www.tug.org/texlive/acquire-netinstall.html
http://anthesphoria.net/FreeBSD/TeXLive-2018/
Comment 29 Kurt Jaeger freebsd_committer 2018-05-12 19:51:51 UTC
So, to proceed here we need a new patch on top of the current tree -- any takers ?
Comment 30 Kurt Jaeger freebsd_committer 2018-05-12 19:57:38 UTC
I've looked at the TL 2018 FreeBSD specific webpage and maybe I'm just not getting it, but how would I use that version ? Which URL to download, how to extract/install ?
Comment 31 Kurt Jaeger freebsd_committer 2018-05-12 20:02:14 UTC
(In reply to Zsolt Udvari from comment #24)
About your suggestions:

- Yes, it would be possible to store the four values in bsd.tex.mk.

- Yes, reducing the number of ports can be done if it helps to speed up the
  porting process.

- If texlive-texmf can be split, fine. It's just difficult to know
  where to split (I have no clue, to be honest)
Comment 32 Michael Osipov 2018-05-12 21:57:24 UTC
(In reply to Kurt Jaeger from comment #30)

It is not that straightforward, it is actually quite complicated. I have deployed TL2017/2018 on our servers and found several bugs in the components.
I also had extensive communication with the FreeBSD maintainer of TL, Nikola Lecic (diacritics missing), on that topic. Here are my findings for the entire community:

Note: I don't like to install software I am not able to compile myself. This puts me into a risk where I cannot patch a bug myself.

1. The installation is handled by a Perl-based installed which also requires wget too and does not use Perl's HTTP client. It is interactive (can be fed by config file) which lets you select the destination and well as the scheme of packages. Those are downloaded from a remote mirror, including precompiled binaries.
2. The precompiled binaries upto 2017 where compiled with FreeBSD 7.2 with ancient port versions. Version 2018 is compiled with 10.0-RELEASE. Nikola exlpains this with the first verion available with LLVM in base. Ports tree is out of date in that version too.
3. All binaries from remote include statically linked dependencies, a busload of. (selfcontained). Unacceptable with the ports system.
4. xelatex is broken in 2017 on i386 with 10.x and 11.x, ICU suffers from  memory alignment errors and fails to init charset converters (akin to iconv). Though, I see no reason to use ICU here in the Font Manager.
5. xelatex in 2018 incorectly links to libfontconfig.so.6 (as far as I remember). Needs to be fixed with ldmap.conf. Nikola is aware of.
6. You can provide a precompiled version of TeXLive base to the installer to avoid pulling natives. Did not try yet.
7. The installation is not selfcontained in the way that the libs and binaries obey PREFIX. I.e., using pdflatex/xelatex/etc call kpathsea not from the same PREFIX, but from PATH. This can lead to a lot of funny things you'd never think about. So you *must* always modify PATH. Basically, you cannot really use several versions in parallel and do /usr/local/texlive/2018/bin/xelatex or /usr/local/texlive/2019/bin/xelatex.
8. The default approach does not install executables to PREFIX/bin, but to PREFIX/bin/{arch}-{os} for whatsoever reason.
9. To have decent binaries, you must compile from source. Though, you can configure a lot. Did so, had a lot of errors at runtime. This stuff seems like magic to me.
10. I did ask how to use my compiled binaries reasonably with the installable schemes, thus instructing the installer to use my binaries. Didn't really a decent answer yet.
11. The build requires gmake, but doesn't really make this clear or how to configure with 'gmake' instead of BSD make.
12. Since ports aren't allowed to download stuff, I really don't know how to forbid tlmgr to download packages from the net. Do we need to prepackage them?
13. Installing additional packages with tlgmr will likely cause daily messages that extraneous files have been added to dirs present in pkg-plist.

For all of these reasons and the way the installation is structured, I would refrain from using such a port. In fact, it does not qualify for the ports system unless someone really puts some time in it.

We'd like/will to put TeXLive along with a Tomcat-based application at the core of our tech documentation system for thousands of users for the months to come, so I have a strong commitment to FreeBSD, but how to solve this problem?

To be honest, I consider most people would be better off with the installer.
Comment 33 Zsolt Udvari 2018-07-18 08:24:30 UTC
I began to create new ports:
- texlive-bin (it will obsolote texlive-base and its devel/tex-* dependencies)
- texlive-texmf packages (splitting smaller ones as described some time ago)
Comment 34 O. Hartmann 2018-07-18 15:14:52 UTC
Thank you very much for working on this matters!

Regards, oh
Comment 35 alessandro 2018-07-21 16:28:52 UTC
(In reply to Zsolt Udvari from comment #33)
Thank you for your effort!
Comment 36 Zsolt Udvari 2018-08-08 10:41:14 UTC
Created attachment 196002 [details]
Texlive 2018 gzipped update (01)
Comment 37 Zsolt Udvari 2018-08-08 10:42:49 UTC
I think the first publishable version is ready.

The diff is here too: http://freebsd.uzsolt.hu/diff/print/texlive-2018/.
Builded packages is here (on 11.2, amd64): http://freebsd.uzsolt.hu/test_pkg/
You can download them and "pkg add ..." (and should remove old tex-* and texlive-* specific packages
(I think I added every conflicts to Makefile but who knows)).

Some background information:

print/texlive-kpathsea:
The search engine (was: devel/tex-kpathsea).
It's better to separate it because the generating format files (*.fmt) is needed it.
And its separation isn't hard and doesn't need any patches.

print/texlive-bin:
The main package which contains the main binaries and some scripts. It contains the generated format files (was: print/tex-formats).
To generate the formats needed (the new) print/texlive-texmf-core. With it can compile some simple tex-files to dvi.
The texlive-bin has options: ALEPH, LUATEX, PDFTEX and XETEX. I think the default LUATEX, PDFTEX and XETEX is enough.
If ports system will support subpackages maybe can split this package.
Don't create separate tex-formats package because can't handle the port options in external port.

====

The monster texlive-texmf package splitted. The main idea is taken
from Arch Linux (https://www.archlinux.org/packages/?q=texlive).
There is DOCS option. The documents are placed in PREFIX/share/doc/texlive-texmf
directory's subdirectories.

print/texlive-texmf-core:
It's needed by print/texlive-bin to create format files.
With this can compile some tex-files to dvi but not enough to LaTeX and not so simple cases.

print/texlive-texmf-base:
I think with this package can compile many TeX and LaTeX files.

print/texlive-texmf-*every*other*:
See their pkg-descr.* :)

The print/texlive-texmf-all is a common ("master", meta) port.
It causes install all texlive-texmf-* packages and provides a minimal "framework" to
minimize the similar works.

===

TODO:
- testing, testing, testing
  I'm using (only) pdflatex and
  don't the others so I can' test them
- add these ports to print/Makefile
- remove Mk/bsd.tex.mk and modify USE_TEX-releated ports
- add DEPRECATED and EXPIRATION_DATE to old texlive-ports
- maybe add dependencies between texlive-texmf-* packages
  if it's needed (imho it would good with user-based testing
  because it's a huge work to check *every* TeX-package's
  dependencies).

IDEAS:
- maybe can update texlive-texmf-* every (1-2-3) months, because the packages are updating
  and appears new packages continuously (see https://ctan.org/ctan-ann).
  It's not too hard because TeXLive has
  * subversion access (https://tug.org/texlive/svn/) to view the logs and
  * rsync access to create a huge texlive-texmf-%monthly%.tar.xz files
  I think can do it.
  In this case the tlmgr is maybe unneeded.
- maybe can create splitted texlive-texmf-*-%version%.tar.xz source files (as Arch does)
  because they would be smaller and faster the build process
  (hasn't read the whole 2G tar.xz file every time).
Comment 38 Zsolt Udvari 2018-08-08 10:48:38 UTC
Review created: https://reviews.freebsd.org/D16623
Comment 39 Michael Osipov 2018-08-08 14:33:22 UTC
Zsolt,

how do you intend to cover the installation schemes the tl-install provides to chose from? I do use medium which is fully sufficient for us.
Comment 40 Zsolt Udvari 2018-08-09 04:28:57 UTC
(In reply to Michael Osipov from comment #39)
I don't know the tl-install (and its method).

You should install texlive-bin and its dependencies and choose from texlive-texmf-* package - based on your using habits :)
I think the texlive-texmf-base is necessary.
If you used to create beamer documents, should install texlive-texmf-presentation; if you're a teacher (as me) maybe the texlive-texmf-school is useful; if you're a musician the texlive-texmf-music is your package :)
Comment 41 Michael Osipov 2018-08-10 12:39:53 UTC
(In reply to Zsolt Udvari from comment #40)
 I checked the patch. It is *massive*. Kudos for the huge work. It is so different to what tl-install does. Especially the port downloads gigabytes (texlive-20180414-texmf.tar.xz) to make stuff available.
Comment 42 Zsolt Udvari 2018-08-10 17:56:34 UTC
(In reply to Michael Osipov from comment #41)
Yes, it's different. TexLive provides this huge file but was my idea (the last in my comment): "maybe can create splitted texlive-texmf-*-%version%.tar.xz source files"

But not so bad the situation: it's enough to download only once :)
Comment 43 Michael Osipov 2018-08-10 18:10:37 UTC
(In reply to Zsolt Udvari from comment #42)

You should really contact Nikola Lecic, he was really helpful in finding a nasty bug in TeXLive on FreeBSD.
Comment 44 Zsolt Udvari 2018-08-17 05:44:01 UTC
https://svn.uzsolt.hu/freebsd-texlive/
Comment 45 marc.priggemeyer 2018-11-03 17:10:14 UTC
Hi, thanks for the effort! I tried to use your packages and encountered the following problems:
- ftp://tug.org restricts the number of connections from one host. Using poudriere with multiple build jails fails for most of the packages.
- otherwise the packages build without issue for FreeBSD 12

Once using the packages (I just installed all of them to see how far I got), pdflatex didn't find biblatex.sty and url.sty to name just two that prevented me from typesetting my .tex source to pdf.

How do I need to understand your intention of separating everything into differnt packages. Should biblatex and url for instance go into a separate package, or should these two be included in one of those you already prepared?

Best regards,
Marc
Comment 46 Zsolt Udvari 2018-11-03 17:15:51 UTC
(In reply to marc.priggemeyer from comment #45)
Did you checkout svn://svn.uzsolt.hu/freebsd-texlive and use it?
Comment 47 marc.priggemeyer 2018-11-03 22:09:17 UTC
(In reply to Zsolt Udvari from comment #46)
Yes, I did. I merged it into my local repository, built all your packages with poudriere and installed them. 
I already checked your pkg-plist files and didn't find biblatex.sty there either.
Comment 48 Zsolt Udvari 2018-11-04 09:51:17 UTC
(In reply to marc.priggemeyer from comment #47)
Updated on svn.uzsolt.hu. Could you test again?

(I don't use biblatex so I didn't notice that this is missed.)
Comment 49 marc.priggemeyer 2018-11-04 14:26:42 UTC
(In reply to Zsolt Udvari from comment #48)
Thanks for the update. Poudriere is currently building your texlive packages again. I will report on the state there later.

I created a differential on reviews for latex-biber, if you do not use biblatex or biber that should not be of interest for you but others might be concerned. It's biber version 2.11 that is compatible with the texlive version you created packages for (20180414), so whoever is willing to try please go ahead.(https://reviews.freebsd.org/D17836)

I also tried installing texlive from scratch (without packages) using the Build script provided in the archives. That worked well an I have a running distribution there. Hopefully and once your texlive packages are built, I will try to make a diff between both versions to see if there are any other things missing.
Comment 50 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-01 07:05:53 UTC
Comment on attachment 196002 [details]
Texlive 2018 gzipped update (01)

Duplicates review
Comment 51 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-01 07:06:48 UTC
Patch (in review D16623) needs updating, to include framework changes, and replaced/merged port removals.
Comment 52 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-01 07:09:46 UTC
This is a framework update as well as individual ports

2019 was released 29 April 2019
Comment 53 Zsolt Udvari 2019-06-01 07:55:50 UTC
(In reply to Kubilay Kocak from comment #52)
I'm working on 2019: https://svn.uzsolt.hu/listing.php?repname=freebsd-texlive
But now I don't have too many time to do it. Maybe one month later will be more.
Comment 54 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-01 08:20:33 UTC
(In reply to Zsolt Udvari from comment #53)

Thanks for the follow-up. Please update the diff in the review after QA testing, and its ready to review
Comment 55 Zsolt Udvari 2019-06-01 08:36:05 UTC
(In reply to Kubilay Kocak from comment #54)
It's not ready yet only the binaries and its dependency. But if it's (almost) ready I'll update of course.