Bug 237332

Summary: Update shells/ksh93 to 2020.0.0-alpha1
Product: Ports & Packages Reporter: Cy Schubert <cy>
Component: Individual Port(s)Assignee: Cy Schubert <cy>
Status: Closed FIXED    
Severity: Affects Only Me CC: arved, crees, fuz, pi, saper, siteshwar
Priority: --- Flags: bugzilla: maintainer-feedback? (saper)
cy: maintainer-feedback?
Version: Latest   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
Proposed new ksh93 port
none
Proposed new ksh93 port
none
Proposed new ksh93 port none

Description Cy Schubert freebsd_committer freebsd_triage 2019-04-17 12:46:24 UTC
With this upstream commit:

commit 96d1960c18f9ea93fde5f999bc5450d636d45163 (tag: 2020.0.0-alpha1)
Author: Siteshwar Vashisht <svashisht@redhat.com>
Date:   Tue Apr 16 19:40:21 2019 +0200

    Update fallback version number to `2020.0.0-alpha1`
    
    When git tags are not available, fallback to `2020.0.0-alpha1` version.

1. Delete shells/ksh93
2. svn copy shells/ksh93-devel to shells/ksh93.
3. Change the tag to 2020.0.0-alpha1.
4. Commit.

This resolves many issues as documented on github.
Comment 1 Siteshwar Vashisht 2019-04-19 13:30:13 UTC
(In reply to Cy Schubert from comment #0)

Thanks a lot for taking care of it.
Comment 2 Siteshwar Vashisht 2019-04-19 13:31:52 UTC
fwiw if you need any help with packaging, don't hesitate to open an issue on GitHub.
Comment 3 Cy Schubert freebsd_committer freebsd_triage 2019-04-19 14:11:45 UTC
No packaging help is need but thanks. My proposal is to use shells/ksh93-devel (maintained by me) as a basis for shells/ksh93. The current maintainer needs to approve. He can maintain the port after the update.
Comment 4 Marcin Cieślak 2019-04-23 13:01:25 UTC
My original intent to take over ksh93 port was to produce a shared library that could be used to build dtksh and drop ancient ksh93 that is bundled with CDE.

I am not yet ready for this change; first I'd need to have a closer look at the 2020 release; also I am not sure it is possible to have a shared library at all with the new mason-based build?
Comment 5 Siteshwar Vashisht 2019-04-23 13:47:47 UTC
(In reply to Marcin Cieślak from comment #4)

In that case, I would suggest you to introduce a separate compatibility package (old ksh) for dtksh in FreeBSD.
Comment 6 Cy Schubert freebsd_committer freebsd_triage 2019-04-23 14:15:15 UTC
Agreed, you cannot hijack a port to maintain it at a backlevel in order to support another port or software. Let's rename the current ksh93 to ksh93-legacy or ksh93v and bring this port up to date. I'm sure this port as it is now has many security bugs that qualify it for FORBIDDEN=.

Look at what I've done with krb5, xpdf, racoon, cfengine, and syslog-ng. For various reasons we maintain current and backlevel ports. But to restrict a port to a certain level because of a dependency is not valid.
Comment 7 Cy Schubert freebsd_committer freebsd_triage 2019-04-25 02:36:05 UTC
As you're interested in building a ksh for cde, you should talk to crees@ about cde. He's the cde port maintainer.

Also, I recall from my Solaris and Tru64 days, dtksh is not the same as ksh. Replacing dtksh with ksh is not a good idea.

Let's simply replace ksh93 with clone of ksh93-devel, pointing to the latest tag that att/ast has cut.
Comment 8 Chris Rees freebsd_committer freebsd_triage 2019-04-25 19:11:39 UTC
I'm going to try building CDE without ksh.  I have a hunch it'll just use the builtin dtksh.
Comment 9 Cy Schubert freebsd_committer freebsd_triage 2019-04-25 19:40:18 UTC
You do need ksh93 or ksh93-devel. It runs some ksh scripts during the build. The only ksh's that work are are one of the att/ast ksh93 ports because these ksh's support the -d option.

I've decided to try building the cde port on my -CURRENT laptop. It fails due to an error flagged by clang. (Sorry I forgot what the error is.) Using gcc8 gets over that hump but unfortunately it also needs libstdc++ but is unable to find the libstdc++ bundled with gcc8, so yet another hack.

Finally cde fails to build some documentation due to a flex error -- it bundles its own flex under a db* name.

I think what the saper@saper.info is trying to do is avoid building ksh again but he fails to realize that dtksh is a different animal. Every system I worked on 20+ years ago with CDE had both. CDE used dtksh while apps, like Oracle and Remedy, used ksh. The O/S (Solaris and Tru64) used Bourne.

IMO it is a mistake to replace dtksh with ksh93. Backporting the diffs to have ksh93 build both is a maintenance nightmare. Whoever commits to this endeavour will subsequently give the next person taking over the port a lot of work to maintain for no gain whatsoever.
Comment 10 Chris Rees freebsd_committer freebsd_triage 2019-04-25 20:11:52 UTC
If you make a patch for the new ksh93 port, I'll happily test it against CDE.
Comment 11 Marcin Cieślak 2019-04-25 20:15:39 UTC
dtksh is ksh93

CDE delivers a bundled ksh93 version, pretty old one and that is harder to get to compile than the current devel/ksh93 port

Currently CDE does not depend on ksh93 being already installed. I would like to introduce that dependency to make building and maintenance of dtksh easier.
Comment 12 Marcin Cieślak 2019-04-25 20:28:43 UTC
(In reply to Marcin Cieślak from comment #11)

Sorry, what I mean it does not depend on ksh93 to bootstrap dtksh. It is a current dependency of the port.
Comment 13 Marcin Cieślak 2019-04-25 20:37:36 UTC
Apart from dtksh, kornshell is used extensively to run .dt actions and other shell scripts. Also CDE installer itself seems to want ksh.

git grep -i kornshell |grep -v sgm | wc -l | less

on the oldish CDE master (66db2259) returns 135 hits.

I hope 2020.0.0-alpha1 could be a drop-in replacement here. It won't probably be one for dtksh, though.
Comment 14 Chris Rees freebsd_committer freebsd_triage 2019-04-25 20:58:12 UTC
(In reply to Marcin Cieślak from comment #13)

You're welcome to take CDE from me at any point :)

I've tested it with ksh93-devel, which unsurprisingly builds fine, but I'll runtime test it with the newer version.
Comment 15 Cy Schubert freebsd_committer freebsd_triage 2019-04-26 01:58:31 UTC
(In reply to Marcin Cieślak from comment #11)
It does depend on ksh93. It uses it before dtksh is built and it references the name ksh93. This is borne out by the upline's on doc, here https://sourceforge.net/p/cdesktopenv/wiki/FreeBSDBuild/.

Also, as I said, I was playing around with it last night. It does use ksh93 before dtksh is built.

dtksh is a little different from ksh93. dtksh is built on top of ksh93 with support for X and Motif APIs. If you do want to separate it out, it must be a new shells/dtksh port separate from ksh93. Why? The X and Motif APIs are not supported by att/ast. Back porting patches every update is unsustainable.

As updating dtksh to ksh93 standards is your goal, update the x11/cde port. Then push those changes back to the cde upline and share your work. It is preferred not to maintain custom patch as much as possible.

I'll attach my proposed ksh93 port to replace shells/ksh93 next comment.
Comment 16 Cy Schubert freebsd_committer freebsd_triage 2019-04-26 02:01:15 UTC
Created attachment 204021 [details]
Proposed new ksh93 port

Apply as a patch to your ports tree.
Comment 17 Cy Schubert freebsd_committer freebsd_triage 2019-04-26 02:04:45 UTC
From the O'Reilly book (which I have here at home):

https://docstore.mik.ua/orelly/unix3/korn/appa_04.htm

dtksh supports the OSF/Motif graphical Toolkit by making its routines available as built-in commands. This allows programmers to combine the Korn shell's strength as a Unix systems programming environment with the power and abstraction of the Toolkit. The result is a unified environment for quick and easy development of graphics-based software.



You cannot simply replace dtksh with ksh93. If you want to update dtksh, either fork only dtksh from cde or (preferred) push your changes back to the cde developers.
Comment 18 Cy Schubert freebsd_committer freebsd_triage 2019-04-26 02:23:00 UTC
(In reply to Chris Rees from comment #14)
Typically CDE doesn't use ksh93. It uses the ksh with builtin X and Motif APIs: dtksh.

The plan going forward should be:

1. Update shells/ksh93 to 2020.0.0-alpha1, having it track 2020.0.0 to release.
2. Backport ksh93 to dtksh, pushing the changes back upstream.
Comment 19 Chris Rees freebsd_committer freebsd_triage 2019-04-26 07:28:53 UTC
No problem, I'll have a go.  By the way, I think you need to use DISTVERSION as that'll give correct ordering if the a means alpha.
Comment 20 Siteshwar Vashisht 2019-04-26 11:26:49 UTC
(In reply to Cy Schubert from comment #16)

I see that you have set version number to `2020.0.0a1`. Will it break upgrade path on FreeBSD ? We had to set epoch on fedora to ensure that package manager will pick up the latest version.
Comment 21 Chris Rees freebsd_committer freebsd_triage 2019-04-26 12:46:46 UTC
DISTVERSION=2020.0.0-alpha1

should generate correct version, and won't need an epoch bump on upgrades.
Comment 22 Cy Schubert freebsd_committer freebsd_triage 2019-04-26 19:41:42 UTC
Created attachment 204047 [details]
Proposed new ksh93 port

This fixes the version number so that 2020.0.0.a1 is older than 2020.0.0. Also removes an unused commit date variable not removed when copied from ksh93-devel.
Comment 23 Cy Schubert freebsd_committer freebsd_triage 2019-04-26 19:48:44 UTC
Created attachment 204048 [details]
Proposed new ksh93 port

DISTVERSION.
Comment 24 Siteshwar Vashisht 2019-05-18 14:53:40 UTC
Any progress on this ?
Comment 25 Cy Schubert freebsd_committer freebsd_triage 2019-05-18 15:28:51 UTC
As we haven't heard from the maintainer, it should be safe to commit the patch I posted here. I'll do that when I get back.
Comment 26 Chris Rees freebsd_committer freebsd_triage 2019-05-18 16:37:33 UTC
x11/cde seems happy with ksh-devel.
Comment 27 commit-hook freebsd_committer freebsd_triage 2019-05-28 02:32:26 UTC
A commit references this bug:

Author: cy
Date: Tue May 28 02:32:15 UTC 2019
New revision: 502844
URL: https://svnweb.freebsd.org/changeset/ports/502844

Log:
  This is the first part of a two part commit to update shells/ksh93 to
  2020.0.0-alpha1.

  Following this commit will be an svn copy shells/ksh93-devel to
  shells/ksh93, replacing the git tag with DISTVERSION 2020.0.0-alpha1.

  PR:		237332
  Approved by:	maintainer timeout

Changes:
  head/shells/ksh93/
Comment 28 commit-hook freebsd_committer freebsd_triage 2019-05-28 02:36:32 UTC
A commit references this bug:

Author: cy
Date: Tue May 28 02:35:40 UTC 2019
New revision: 502845
URL: https://svnweb.freebsd.org/changeset/ports/502845

Log:
  This is the second part of a two part commit Updating shells/ksh93 to
  2020.0.0-alpha1.

  This commit:

  1. Deletes shells/ksh93, completed by r502844.
  2. svn copies shells/ksh93-devel to shells/ksh93 -- this commit.
  3. Replaces the git tag with DISTVERSION 2020.0.0-alpha1 -- this commit.

  PR:		237332
  Reviewed by:	crees@
  Approved by:	maintainer timeout
  Reminded by:	Siteshwar Vashisht <svashisht@redhat.com>
  		(ksh93 upstream maintainer)

Changes:
  head/shells/ksh93/
  head/shells/ksh93/Makefile
  head/shells/ksh93/distinfo
Comment 29 Cy Schubert freebsd_committer freebsd_triage 2019-05-28 02:37:30 UTC
Update is complete.
Comment 30 Marcin Cieślak 2019-05-28 08:30:29 UTC
Cy,

I told you I do not approve this, therefore please remove my email as the maintainer of the new port.

saper
Comment 31 Cy Schubert freebsd_committer freebsd_triage 2019-05-28 11:55:00 UTC
All that you said was "I am not yet ready for this change; first I'd need to have a closer look at the 2020 release; also I am not sure it is possible to have a shared library at all with the new mason-based build?". You did not say why you needed a shared library expect to say that you wanted to replace dtksh in CDE with ksh. I specifically told you you cannot replace dtksh with ksh because dtksh included Motif and X functions and primitives for shell programming.

My point was to replace dtksh with ksh93, which itself is a subset of dtksh was wrong. You cannot do this.

Additionally one of our upline ksh93 maintainer asked if there was any progress on this. It has been at least two weeks since my request what the plan was, therefore maintainer timeout rules applied.

Please articulate what it is you are trying to do. I am confused as to what your plan is.

Your name can be removed from maintaining the port but this is not preferred. I'd rather not do this, I'd like to see you continue as maintainer but I also want to understand your plan.
Comment 32 Siteshwar Vashisht 2019-05-28 12:20:32 UTC
> I told you I do not approve this, therefore please remove my email as the maintainer of the new port.

As I understand, Martin is primarily interested in ksh due to CDE. We have ripped out lot of code from ast codebase (dtksh etc.) and retained only core ksh code for ksh-2020.0.0. That's why he is not interested in adopting this port, despite the fact that new version of ksh still works well with CDE (as noted by Chris in comment 26).

There is a difference between priorities of current upstream maintainers and some old users of ksh. It's no secret that this project was in a very unmaintainable state and moving forward required breaking some existing use cases. We value every member of ksh community and it makes me sad that some users are unhappy with our decisions. Unfortunately we are bound to live with this situation.
Comment 33 Cy Schubert freebsd_committer freebsd_triage 2019-05-28 12:52:31 UTC
(In reply to Siteshwar Vashisht from comment #32)

What do you suggest the path forward for Martin is? A possible dtksh port? If a dtksh port, based on what though?
Comment 34 Cy Schubert freebsd_committer freebsd_triage 2019-05-28 12:53:49 UTC
Martin, if you want I can resurrect the deleted ksh93 to dtksh, ksh93-dtksh, ksh93-legacy or some other name. This will allow you the opportunity to develop something unique to your needs.
Comment 35 Siteshwar Vashisht 2019-05-28 12:57:00 UTC
> What do you suggest the path forward for Martin is? A possible dtksh port? If a dtksh port, based on what though?


I made my suggestion in comment 5. Create a separate compatibility package for older version of ksh and keep it under low maintenance.
Comment 36 Marcin Cieślak 2019-05-28 16:13:29 UTC
Thank you Siteshwar - this sums up my motivation better than I would!

I was going to explore the possibility to use ksh93 as an embedded library (like tcl or lua) - one of the original design goals of the Korn Shell. I got it working for some simple scripts and I was pretty sure this could be done to provide ksh93 as a library runtime dependency so that dtksh can be linked against this.

This is an interesting toy project that has nothing to do with Solaris or RedHat users that need a stable ksh93 _right_now_ and therefore I fully understand things that I want to play with are out of scope for the current "alpha" project and that's perfectly fine. Different goals, needs, different values and directions of development.

I still continue to use ksh93v as my primary interactive shell. If the alpha 2020 project gets the filename completion fixed, even better :)

I only hope that 2020 will stay 100% compatible with David Korn's book. If that breaks, we can all just switch to bash.
Comment 37 Cy Schubert freebsd_committer freebsd_triage 2019-05-28 18:48:39 UTC
(In reply to Marcin Cieślak from comment #36)
Marcin, If you would like we can bring back any previous version of a port to a ksh93-legacy port. Would you like that instead? What would you like it to be named?
Comment 38 Siteshwar Vashisht 2019-05-28 19:27:00 UTC
> I still continue to use ksh93v as my primary interactive shell. If the alpha 2020 project gets the filename completion fixed, even better :)


Sure. I agree that interactive capabilities of ksh should be improved.

> I only hope that 2020 will stay 100% compatible with David Korn's book. If that breaks, we can all just switch to bash.


This book was released more than 2 decades ago and I am not sure how compatible was last stable version (ksh93u+) with it. But I agree that we should use last stable version of ksh that came out from Bell labs as the reference and any deviation from it needs solid justification.
Comment 39 Cy Schubert freebsd_committer freebsd_triage 2019-05-29 01:35:12 UTC
We can resurrect ksh93v as ksh93-legacy from r499547. Does this sound reasonable?
Comment 40 Marcin Cieślak 2019-05-30 21:29:47 UTC
Thanks. Can we name it shells/ast-ksh please? I hope the SVN port history will be available there as well?
Comment 41 commit-hook freebsd_committer freebsd_triage 2019-06-01 18:32:54 UTC
A commit references this bug:

Author: cy
Date: Sat Jun  1 18:32:41 UTC 2019
New revision: 503248
URL: https://svnweb.freebsd.org/changeset/ports/503248

Log:
  Resurrect the previous shells/ksh93 as  shells/ast-ksh, a ksh93 port
  that the maintainer wishes to use to create a shared library for use
  with other applications such as CDE. It is based on ksh93v (2014-12-24)
  and is incompatible with the direction that att/ast is taking the
  official ksh93 implementation.

  PR:		237332
  Requested by:	maintainer (saper@saper.info)

Changes:
  head/shells/Makefile
  head/shells/ast-ksh/
  head/shells/ast-ksh/Makefile
  head/shells/ast-ksh/distinfo
Comment 42 commit-hook freebsd_committer freebsd_triage 2019-06-01 18:37:01 UTC
A commit references this bug:

Author: cy
Date: Sat Jun  1 18:36:03 UTC 2019
New revision: 503249
URL: https://svnweb.freebsd.org/changeset/ports/503249

Log:
  Fix version going backward due to update to 2020.0.0.a1 (pointy hat to
  yours truly).

  The maintainer no longer wants to maintain ksh93 as his desire is to
  maintain a backlevel port of ksh93 in order to build and support a
  shared library for use by legacy applications, which is inconsistent
  with the direction of the att/ast team on github. I will maintain the
  port.

  PR:		237332
  PR:		238266
  Approved by:	maintiner (saper@saper.info)

Changes:
  head/shells/ksh93/Makefile