Bug 204577

Summary: [NEW MODULE] Mk/Uses/nodejs.mk Framework for NodeJS projects/ports
Product: Ports & Packages Reporter: Yuri Victorovich <yuri>
Component: Ports FrameworkAssignee: Port Management Team <portmgr>
Status: New ---    
Severity: Affects Many People CC: 0mp, adamw, koobs, lifanov, olivierd, ports-bugs, ports
Priority: --- Keywords: needs-patch, needs-qa
Version: Latest   
Hardware: Any   
OS: Any   
Bug Depends on: 204904, 204905    
Bug Blocks: 204578, 204906, 204932, 205053, 205699, 204907    
Attachments:
Description Flags
patch
none
patch adding include statement
none
patch
none
patch
none
patch
none
patch
none
patch
none
Updated patch
none
patch none

Description Yuri Victorovich freebsd_committer 2015-11-16 02:08:30 UTC
Created attachment 163164 [details]
patch

This module is to support NodeJS ports, see dependencies. (ex. EtherPad, EtherCalc, etc)

I am creating this bug report in order for the attached patch to be possibly used by those who wants to use EtherPad. However, there are few NodeJS bugs that need to be resolved until this can be committed. Without this such NodeJS-based ports don't pass poudriere build, and have to have NO_CHECKSUM=yes set, but still can be installed and used locally.
Comment 1 Yuri Victorovich freebsd_committer 2015-11-16 02:12:58 UTC
Created attachment 163166 [details]
patch adding include statement
Comment 2 Baptiste Daroussin freebsd_committer 2015-11-16 09:54:20 UTC
Please do not make yet another bsd.*.mk it should be done as a USES: USES=node or something like that.

Second we do need checksum and make sure we validate that what we are packaging is really what has been tested (aka the sources) so this cannot be added until we can get checksums. so this will have to wait for https://github.com/npm/npm/issues/10406 to be fixed (as pointed in your patch)
Comment 3 Yuri Victorovich freebsd_committer 2015-11-16 10:39:13 UTC
Agreed.
Comment 4 Yuri Victorovich freebsd_committer 2015-11-16 10:41:51 UTC
I will update all patches, as USES=node, once all issues are resolved. So far let people try like this if anybody wants.
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2015-11-16 16:47:05 UTC
@Yuri, also FWIW, this is probably the perfect opportunity and candidate for a review (reviews.freebsd.org)
Comment 6 Yuri Victorovich freebsd_committer 2015-11-17 06:43:12 UTC
Created attachment 163232 [details]
patch
Comment 7 Yuri Victorovich freebsd_committer 2015-11-17 06:51:52 UTC
@Kubilay In reviews.freebsd.org, do I create "Pholio Mock", "Conpherence Room", or "Paste"?
Comment 8 Yuri Victorovich freebsd_committer 2015-11-17 07:30:57 UTC
https://reviews.freebsd.org/P88
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2015-11-17 09:09:54 UTC
(In reply to yuri from comment #8)

None of the above. Create a 'Differential Revision' via 'Differential' -> 'Create Diff', but its preferable to use the Arcanist arc(8) client (using the diff --create command)

For instructions see: https://wiki.freebsd.org/CodeReview
Comment 10 Yuri Victorovich freebsd_committer 2015-11-30 07:34:48 UTC
Created attachment 163675 [details]
patch

Update of USES=nodejs.

Now it only suffers from one problem: volatility of package.json in downloaded packages. Therefore NO_CHECKSUM=yes still has to be used for now and it can't be checked in. The second problem (attempts to connect during build) is fixed.

Now packages build fine in poudriere.

Two ports are currently supported and are completely usable:
* www/etherpad-lite
* www/webtorrent

Volatility of package.json is a bug, and isn't something I can fix - it should be fixed by NodeJS folks. https://github.com/npm/npm/issues/10406
Comment 11 Mathieu Arnold freebsd_committer 2015-11-30 10:21:54 UTC
What is fakegit for ? Can't you add a dependency on git ? Or use USE_GITHUB.

You're not supposed to touch WRKDIR and write files there in the fetch phase, during that stage, you fetch stuff into DISTDIR Then, during the extract phase, you extract/copy stuff from DISTDIR into WRKDIR.

Also, you cannot be using NO_CHECKSUM.

Don't add targets like this : "post-fetch: userprofile_fetch fakegit" or "do-build: npm-build npm-prune". Add them to _USES_build, you can find examples of _USES_<stage> in other USES.
Comment 12 Yuri Victorovich freebsd_committer 2015-11-30 10:40:49 UTC
fakegit is a shell script that delivers files previously downloaded by USE_GITHUB to npm. It itself doesn't depend on git.
Comment 13 Yuri Victorovich freebsd_committer 2015-11-30 10:51:44 UTC
> You're not supposed to touch WRKDIR and write files ...

I only used it for temporary things. Will move them.
Comment 14 Mathieu Arnold freebsd_committer 2015-11-30 11:04:32 UTC
(In reply to yuri from comment #13)
> > You're not supposed to touch WRKDIR and write files ...
> 
> I only used it for temporary things. Will move them.

Note that you are not allowed to write anywhere else :-)
Comment 15 Yuri Victorovich freebsd_committer 2015-11-30 11:13:01 UTC
> Note that you are not allowed to write anywhere else :-)

You should suggest this as an additional check for poudriere, because it is quite happy with such writes. :-)

I will move temp files to ${DISTDIR}/${PORTNAME}.tmp, and will use _USES_fetch.
Comment 16 Olivier Duchateau freebsd_committer 2015-11-30 18:20:56 UTC
@Yuri, I'm also working on Node.js framework, see the latest status report [1]. I use rather the NPM registry by default (it's more simple, because modules are already in JavaScript).
My repository contains almost 300 new ports. I can also build C/C++ extensions, it supports man pages and Node.js scripts (framework automatically create symlink into ${PREFIX}/bin for convenience).
pkg-plist is automatically created, (often there're only 2 or 3 files).

[1] https://www.freebsd.org/news/status/report-2015-07-2015-09.html#Node.js-Modules
Comment 17 Mathieu Arnold freebsd_committer 2015-12-01 11:04:49 UTC
(In reply to Olivier Duchateau from comment #16)
> @Yuri, I'm also working on Node.js framework, see the latest status report
> [1]. I use rather the NPM registry by default (it's more simple, because
> modules are already in JavaScript).
> My repository contains almost 300 new ports. I can also build C/C++
> extensions, it supports man pages and Node.js scripts (framework
> automatically create symlink into ${PREFIX}/bin for convenience).
> pkg-plist is automatically created, (often there're only 2 or 3 files).

It would be great if this patch could make its way into a review, or a PR here, so that other can comment on it.
Comment 18 Yuri Victorovich freebsd_committer 2015-12-01 12:07:54 UTC
@Olivier, When I googled and searched here on bugs@ I couldn't find your work.
You seem to try to make ports for small NodeJS packages like 'Request'. Single NodeJS process can contain many versions of the same package, it is generally impossible to make FreeBSD ports for such small NodeJS packages that would fit all uses. People also tend to update them frequently, so it is impossible to chase such things with separate ports. node-gyp also has flavors, and many use node-pre-gyp instead. It is also easy to fork your own node-gyp and also to include it into the distro. This is the nature of NodeJS - it is very volatile and fickle. Doesn't really match FreeBSD ports philosophy well.

I chose an approach to make the separate FreeBSD ports for large apps that end-users care about. Since npm itself installs modules in a very fragile way, the goal of my USES=nodejs project are:
1. Have framework that allows to relatively easily create ports for large apps
2. Such ports behave nearly identical to all other ports (no hacks like pre-built and pre-packaged ports, ex. npm)
3. One port per app is created, so it is easy to maintain (it would be a hell to maintain separate ports for tiny NodeJS packages)
4. Ports build in a stable and deterministic way
5. Have reasonable delivery time (several NodeJS apps already work fine through ports)

From what I myself can see, project meets these goals quite well. There is only one item, #4, that is impeded by the npm-shrinkwrap bug that causes the requirement to have NO_CHECKSUM=yes. Latest npm version might have fixed this (but we don't have it yet).

Mathieu expressed some valid criticisms, I have already fixed these points in my version, they are really minor. I am very open to hear any other criticisms. This only helps to improve.

Currently I have these ports working and usable:
* www/etherpad-lite
* net-p2p/webtorrent
* devel/grunt
* lang/coffeescript
* www/sharelatex
Comment 19 Olivier Duchateau freebsd_committer 2015-12-01 17:24:15 UTC
(In reply to Mathieu Arnold from comment #17)

@Mathieu, it's still work in progress. It's planned, but not before support of Grunt (almost finished).

(In reply to yuri from comment #18)

@Yuri, my approach is equivalent to npm -g (modules are installed globally). I use same way as Ruby's gems, one module = one port (I respect dependencies found in package.json). I don't understand your comment, because I don't respect ports philosophy. With my framework, users can define do-install, post-install targets if he/her wants (see textproc/node-JSONSelect [1] port for example).
Modules are located into ${LOCALBASE}/lib/node_modules (it's convention for NODE_PATH). 

You can find complete list here [2], and there're some new modules especially SocketIO, jison.

Debian and Fedora also follow same approach as me.

About my Mk/Uses/nodegyp.mk, it is independent, it builds only C/C++ extension, users must define do-install target (except if your module comes from NPM registry, I defined "build" argument in Mk/Uses/node.mk for convenience).

[1] https://www.assembla.com/spaces/cozycloud/subversion/source/HEAD/trunk/textproc/node-JSONSelect
[2] https://people.freebsd.org/~olivierd/nodejs-ports.txt
Comment 20 Olivier Duchateau freebsd_committer 2015-12-01 17:27:35 UTC
And I forget, I don't need hidden files.
Comment 21 Yuri Victorovich freebsd_committer 2015-12-01 18:12:56 UTC
I just couldn't get stable results with 'npm -g install <package>'. Builds for various projects tend to fail with compilation errors, even on Arch linux. Then people need to go to various forums, ask, and usually find that there is some bad update for some NodeJS project, or it is of too old version. I try to make things predictable, when installed package always works the same way.
Comment 22 Torsten Zühlsdorff 2015-12-03 10:17:14 UTC
(In reply to yuri from comment #18)

> I chose an approach to make the separate FreeBSD ports for large apps that 
> end-users care about.

I understand your approach, but there are also ports depending on single small libs. deskutils/ladon for example could be unbroken when having async, glob, mkdirp and yargs in tree.

While large apps are great for end-users we should not forget there is a growing number of tools which npm-dependencies.
Comment 23 Yuri Victorovich freebsd_committer 2015-12-03 10:49:35 UTC
> I understand your approach, but there are also ports depending on single small libs. deskutils/ladon for example could be unbroken when having async, glob, mkdirp and yargs in tree.

> While large apps are great for end-users we should not forget there is a growing number of tools which npm-dependencies.

Ports can be created for the smaller packages too. I mostly meant that large package shouldn't be split into hundreds of small dependencies from ports, like this is done with C/C++ projects.
Comment 24 Yuri Victorovich freebsd_committer 2015-12-04 00:35:40 UTC
@Torsten (In reply to yuri from comment #23)

deskutils/ladon is trivially converted to the building and working port with USES=nodejs, and I suspect other ports you mentioned are the the same way.
Comment 25 Yuri Victorovich freebsd_committer 2015-12-06 10:01:29 UTC
Created attachment 163883 [details]
patch

Update
Comment 26 Olivier Duchateau freebsd_committer 2015-12-06 16:49:50 UTC
(In reply to yuri from comment #25)

I don't understand, why you persist to use Github? If useful module is hosted an another host (and use another revision control system), user must re-define npm-fetch target.

I think it's more simple to use the npm registry website, moreover tarballs are well formed and name is generic. So it can be generalized in framework.

I do not agree to include node.js submodules. I prefer to have one port per module.

IMHO, Grunt and CoffeeScript should be placed in independent files (for example Mk/Uses/grunt.mk and Mk/Uses/coffeescript.mk), loading right variables environment in order to configure and build packages.
Comment 27 Yuri Victorovich freebsd_committer 2015-12-06 17:07:42 UTC
GitHub artifact is only hardcoded in one small place and I will remove it. Individual projects happen to be hosted on GitHub - this is their choice.

However, notion of GitHub is embedded in npm design, please see the the command 'npm install github-download'. Git is also in the specification of package.json, which supports the git clause.

USES=nodejs downloads everything from npm registry, unless git is specified in package.json as a source, in which case git is used.

> I do not agree to include node.js submodules. I prefer to have one port per module.

There are hundreds of modules, and they can be, and are present in different versions within the same process. This is the major feature of NodeJS that it supports this. I don't see how can they correspond to the ports in such situation. Also it is hard to imagine who is going to maintain all the modules. Ease of port creation and maintenance is a major factor here.

> IMHO, Grunt and CoffeeScript should be placed in independent files (for example Mk/Uses/grunt.mk and Mk/Uses/coffeescript.mk), loading right variables environment in order to configure and build packages.

Maybe we will do this. Currently this is only the initial version that only has 5 ports.
Comment 28 Yuri Victorovich freebsd_committer 2015-12-06 17:18:49 UTC
USES=nodejs also runs projects in a very similar way to how they are suggested to be run by developers. They are normally just run from the directory, and don't even define any installation procedure.
Comment 29 Yuri Victorovich freebsd_committer 2015-12-10 11:18:00 UTC
Created attachment 164037 [details]
patch

Now all ports reliably build in poudriere with checksums.
I added the workflow description and fixed most problems.

www/etherpad-lite
net-p2p/webtorrent
devel/grunt
lang/coffeescript
print/sharelatex
Comment 30 Yuri Victorovich freebsd_committer 2015-12-11 09:42:12 UTC
Created attachment 164105 [details]
patch

Update with many improvements.
Comment 31 Torsten Zühlsdorff 2015-12-14 16:10:20 UTC
(In reply to yuri from comment #27)

> There are hundreds of modules, and they can be, and are present in different 
> versions within the same process. This is the major feature of NodeJS that it 
> supports this.

And it is a major disadvantage. I discovered multiple projects which could not build anymore, because the old versions were no longer provided!

We have 2 different approaches here and 2 persons which do much work to get their way working. That is great! Thank you both for your efforts!

But i think we should discuss the advantages and disadvantages of both solutions and try to find the way we want.

From my point of view:

There are many different modules and it is a major task to make single ports for them. This is an error prone but working way, as rubygems teached us. But this as the advantage that everything is clear and known

Creating bundles saves as the port managing, which is a huge advantage. But we need a way to guarantee that everything is know before installing (no fetching of new libs after fetch-phase). Also how to you handle multiple usage of the same library? Multiple installation, references, etc? Currently it is not clear how do you solve it. Can you please explain this to me?
Comment 32 Yuri Victorovich freebsd_committer 2015-12-14 21:11:17 UTC
(In reply to Torsten Zühlsdorff from comment #31)

nodejs.mk fetches everything needed for the port during the fetch phase, then packages port with all dependencies so that this port always works the same way. Build doesn't require the network access. We use shrinkwrap feature that npm provides and recommends to use to freeze the dependencies.

I just don't see how can individual ports be created for all dependencies. This is a very large amount of work to maintain them. Also please note that different projects can require different versions. How can having a single version on ports suit such requirement? Also if the older distfile disappears on the source, this will break the individual ports the same way. So besides saving the space having them doesn't provide any advantage in NodeJS case.

I currently have 5 ports working, and looking at some other, larger projects to make ports for. Making ports for simple projects is trivial, finding larger projects that can also be built on FreeBSD seems to be a challenge (few of the ones that I found require large external C++ projects to work that weren't ported or require qt-5.5).
Comment 33 Yuri Victorovich freebsd_committer 2015-12-27 14:14:55 UTC
Created attachment 164721 [details]
Updated patch

Made shrinkwrap deterministic.
Comment 34 Yuri Victorovich freebsd_committer 2015-12-28 06:39:13 UTC
Created attachment 164745 [details]
patch

Implemented auto-plist.
Comment 35 Yuri Victorovich freebsd_committer 2016-12-22 16:07:53 UTC
Will finish this once 9 is out.
Comment 36 Adam Weinberger freebsd_committer 2016-12-22 16:13:37 UTC
Bug #213947 is doing the same thing. Can you and Bradley combine your efforts?
Comment 37 Olivier Duchateau freebsd_committer 2016-12-22 18:07:04 UTC
(In reply to Adam Weinberger from comment #36)

I'm also working in Nodejs infrastructure, https://www.freebsd.org/news/status/report-2015-10-2015-12.html#Node.js-Modules
Comment 38 Nikolai Lifanov freebsd_committer 2017-02-18 23:50:15 UTC
Hi!

This is blocking some other work. Are there any news on updated NodeJS infra?
Comment 39 Mathieu Arnold freebsd_committer 2017-02-21 12:39:01 UTC
There were a few competing versions of a USES=node thing, so maybe Olivier could pitch in.

(Also, the current code, here, is pretty unreadable, and completely undocumented)
Comment 40 Olivier Duchateau freebsd_committer 2017-02-21 16:56:50 UTC
In the past I worked on NodeJS USES framework. (my repo [1] is outdated, I'm no longer interested to work on it, feel free to clone, update or whatever).

You can find a Draft [2] of Porter's Handbook, and a complete list [3] of ports available.

A NodeJS module (not extension, which are often written in C or C++) must contains package.json and one or several JavaScript files that's all. My USES framework tries to follow this guideline.

Not mentioned in documentation page, but it supports DEFAULT_VERSION variable  (available in /etc/make.conf). Module are downloaded from the NPM registry, and pkg-plist is build automatically. Users can override main targets (without to be in conflict with targets of framework).

[1] https://app.assembla.com/spaces/cozycloud/subversion/source
[2] https://people.freebsd.org/~olivierd/porters-handbook/uses-nodejs.html
[3] https://people.freebsd.org/~olivierd/nodejs-ports.txt
Comment 41 Nikolai Lifanov freebsd_committer 2017-03-02 17:30:39 UTC
Is this ready to move forward?
It's blocking a few other PRs.
Comment 42 Mathieu Arnold freebsd_committer 2017-03-02 22:43:10 UTC
In its current state, it is overly complicated and quite unreadable, so, no, it is not ready to move forward.