Bug 204577
| Summary: | [NEW MODULE] Mk/Uses/nodejs.mk Framework for NodeJS projects/ports | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Yuri Victorovich <yuri> | ||||||||||||||||||||
| Component: | Ports Framework | Assignee: | Port Management Team <portmgr> | ||||||||||||||||||||
| Status: | Closed Overcome By Events | ||||||||||||||||||||||
| Severity: | Affects Many People | CC: | 0mp, adamw, koobs, lifanov, olivierd, ports-bugs, ports, swills, tcberner | ||||||||||||||||||||
| Priority: | --- | Keywords: | needs-patch, needs-qa | ||||||||||||||||||||
| Version: | Latest | ||||||||||||||||||||||
| Hardware: | Any | ||||||||||||||||||||||
| OS: | Any | ||||||||||||||||||||||
| Bug Depends on: | 204904, 204905 | ||||||||||||||||||||||
| Bug Blocks: | 204932, 205053, 204578, 204906, 204907, 205699 | ||||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||||
Created attachment 163166 [details]
patch adding include statement
Please do not make yet another bsd.*.mk it should be done as a USES: USES=node or something like that. Second we do need checksum and make sure we validate that what we are packaging is really what has been tested (aka the sources) so this cannot be added until we can get checksums. so this will have to wait for https://github.com/npm/npm/issues/10406 to be fixed (as pointed in your patch) Agreed. I will update all patches, as USES=node, once all issues are resolved. So far let people try like this if anybody wants. @Yuri, also FWIW, this is probably the perfect opportunity and candidate for a review (reviews.freebsd.org) Created attachment 163232 [details]
patch
@Kubilay In reviews.freebsd.org, do I create "Pholio Mock", "Conpherence Room", or "Paste"? (In reply to yuri from comment #8) None of the above. Create a 'Differential Revision' via 'Differential' -> 'Create Diff', but its preferable to use the Arcanist arc(8) client (using the diff --create command) For instructions see: https://wiki.freebsd.org/CodeReview Created attachment 163675 [details] patch Update of USES=nodejs. Now it only suffers from one problem: volatility of package.json in downloaded packages. Therefore NO_CHECKSUM=yes still has to be used for now and it can't be checked in. The second problem (attempts to connect during build) is fixed. Now packages build fine in poudriere. Two ports are currently supported and are completely usable: * www/etherpad-lite * www/webtorrent Volatility of package.json is a bug, and isn't something I can fix - it should be fixed by NodeJS folks. https://github.com/npm/npm/issues/10406 What is fakegit for ? Can't you add a dependency on git ? Or use USE_GITHUB. You're not supposed to touch WRKDIR and write files there in the fetch phase, during that stage, you fetch stuff into DISTDIR Then, during the extract phase, you extract/copy stuff from DISTDIR into WRKDIR. Also, you cannot be using NO_CHECKSUM. Don't add targets like this : "post-fetch: userprofile_fetch fakegit" or "do-build: npm-build npm-prune". Add them to _USES_build, you can find examples of _USES_<stage> in other USES. fakegit is a shell script that delivers files previously downloaded by USE_GITHUB to npm. It itself doesn't depend on git. > You're not supposed to touch WRKDIR and write files ...
I only used it for temporary things. Will move them.
(In reply to yuri from comment #13) > > You're not supposed to touch WRKDIR and write files ... > > I only used it for temporary things. Will move them. Note that you are not allowed to write anywhere else :-) > Note that you are not allowed to write anywhere else :-)
You should suggest this as an additional check for poudriere, because it is quite happy with such writes. :-)
I will move temp files to ${DISTDIR}/${PORTNAME}.tmp, and will use _USES_fetch.
@Yuri, I'm also working on Node.js framework, see the latest status report [1]. I use rather the NPM registry by default (it's more simple, because modules are already in JavaScript).
My repository contains almost 300 new ports. I can also build C/C++ extensions, it supports man pages and Node.js scripts (framework automatically create symlink into ${PREFIX}/bin for convenience).
pkg-plist is automatically created, (often there're only 2 or 3 files).
[1] https://www.freebsd.org/news/status/report-2015-07-2015-09.html#Node.js-Modules
(In reply to Olivier Duchateau from comment #16) > @Yuri, I'm also working on Node.js framework, see the latest status report > [1]. I use rather the NPM registry by default (it's more simple, because > modules are already in JavaScript). > My repository contains almost 300 new ports. I can also build C/C++ > extensions, it supports man pages and Node.js scripts (framework > automatically create symlink into ${PREFIX}/bin for convenience). > pkg-plist is automatically created, (often there're only 2 or 3 files). It would be great if this patch could make its way into a review, or a PR here, so that other can comment on it. @Olivier, When I googled and searched here on bugs@ I couldn't find your work. You seem to try to make ports for small NodeJS packages like 'Request'. Single NodeJS process can contain many versions of the same package, it is generally impossible to make FreeBSD ports for such small NodeJS packages that would fit all uses. People also tend to update them frequently, so it is impossible to chase such things with separate ports. node-gyp also has flavors, and many use node-pre-gyp instead. It is also easy to fork your own node-gyp and also to include it into the distro. This is the nature of NodeJS - it is very volatile and fickle. Doesn't really match FreeBSD ports philosophy well. I chose an approach to make the separate FreeBSD ports for large apps that end-users care about. Since npm itself installs modules in a very fragile way, the goal of my USES=nodejs project are: 1. Have framework that allows to relatively easily create ports for large apps 2. Such ports behave nearly identical to all other ports (no hacks like pre-built and pre-packaged ports, ex. npm) 3. One port per app is created, so it is easy to maintain (it would be a hell to maintain separate ports for tiny NodeJS packages) 4. Ports build in a stable and deterministic way 5. Have reasonable delivery time (several NodeJS apps already work fine through ports) From what I myself can see, project meets these goals quite well. There is only one item, #4, that is impeded by the npm-shrinkwrap bug that causes the requirement to have NO_CHECKSUM=yes. Latest npm version might have fixed this (but we don't have it yet). Mathieu expressed some valid criticisms, I have already fixed these points in my version, they are really minor. I am very open to hear any other criticisms. This only helps to improve. Currently I have these ports working and usable: * www/etherpad-lite * net-p2p/webtorrent * devel/grunt * lang/coffeescript * www/sharelatex (In reply to Mathieu Arnold from comment #17) @Mathieu, it's still work in progress. It's planned, but not before support of Grunt (almost finished). (In reply to yuri from comment #18) @Yuri, my approach is equivalent to npm -g (modules are installed globally). I use same way as Ruby's gems, one module = one port (I respect dependencies found in package.json). I don't understand your comment, because I don't respect ports philosophy. With my framework, users can define do-install, post-install targets if he/her wants (see textproc/node-JSONSelect [1] port for example). Modules are located into ${LOCALBASE}/lib/node_modules (it's convention for NODE_PATH). You can find complete list here [2], and there're some new modules especially SocketIO, jison. Debian and Fedora also follow same approach as me. About my Mk/Uses/nodegyp.mk, it is independent, it builds only C/C++ extension, users must define do-install target (except if your module comes from NPM registry, I defined "build" argument in Mk/Uses/node.mk for convenience). [1] https://www.assembla.com/spaces/cozycloud/subversion/source/HEAD/trunk/textproc/node-JSONSelect [2] https://people.freebsd.org/~olivierd/nodejs-ports.txt And I forget, I don't need hidden files. I just couldn't get stable results with 'npm -g install <package>'. Builds for various projects tend to fail with compilation errors, even on Arch linux. Then people need to go to various forums, ask, and usually find that there is some bad update for some NodeJS project, or it is of too old version. I try to make things predictable, when installed package always works the same way. (In reply to yuri from comment #18) > I chose an approach to make the separate FreeBSD ports for large apps that > end-users care about. I understand your approach, but there are also ports depending on single small libs. deskutils/ladon for example could be unbroken when having async, glob, mkdirp and yargs in tree. While large apps are great for end-users we should not forget there is a growing number of tools which npm-dependencies. > I understand your approach, but there are also ports depending on single small libs. deskutils/ladon for example could be unbroken when having async, glob, mkdirp and yargs in tree. > While large apps are great for end-users we should not forget there is a growing number of tools which npm-dependencies. Ports can be created for the smaller packages too. I mostly meant that large package shouldn't be split into hundreds of small dependencies from ports, like this is done with C/C++ projects. @Torsten (In reply to yuri from comment #23) deskutils/ladon is trivially converted to the building and working port with USES=nodejs, and I suspect other ports you mentioned are the the same way. Created attachment 163883 [details]
patch
Update
(In reply to yuri from comment #25) I don't understand, why you persist to use Github? If useful module is hosted an another host (and use another revision control system), user must re-define npm-fetch target. I think it's more simple to use the npm registry website, moreover tarballs are well formed and name is generic. So it can be generalized in framework. I do not agree to include node.js submodules. I prefer to have one port per module. IMHO, Grunt and CoffeeScript should be placed in independent files (for example Mk/Uses/grunt.mk and Mk/Uses/coffeescript.mk), loading right variables environment in order to configure and build packages. GitHub artifact is only hardcoded in one small place and I will remove it. Individual projects happen to be hosted on GitHub - this is their choice. However, notion of GitHub is embedded in npm design, please see the the command 'npm install github-download'. Git is also in the specification of package.json, which supports the git clause. USES=nodejs downloads everything from npm registry, unless git is specified in package.json as a source, in which case git is used. > I do not agree to include node.js submodules. I prefer to have one port per module. There are hundreds of modules, and they can be, and are present in different versions within the same process. This is the major feature of NodeJS that it supports this. I don't see how can they correspond to the ports in such situation. Also it is hard to imagine who is going to maintain all the modules. Ease of port creation and maintenance is a major factor here. > IMHO, Grunt and CoffeeScript should be placed in independent files (for example Mk/Uses/grunt.mk and Mk/Uses/coffeescript.mk), loading right variables environment in order to configure and build packages. Maybe we will do this. Currently this is only the initial version that only has 5 ports. USES=nodejs also runs projects in a very similar way to how they are suggested to be run by developers. They are normally just run from the directory, and don't even define any installation procedure. Created attachment 164037 [details]
patch
Now all ports reliably build in poudriere with checksums.
I added the workflow description and fixed most problems.
www/etherpad-lite
net-p2p/webtorrent
devel/grunt
lang/coffeescript
print/sharelatex
Created attachment 164105 [details]
patch
Update with many improvements.
(In reply to yuri from comment #27) > There are hundreds of modules, and they can be, and are present in different > versions within the same process. This is the major feature of NodeJS that it > supports this. And it is a major disadvantage. I discovered multiple projects which could not build anymore, because the old versions were no longer provided! We have 2 different approaches here and 2 persons which do much work to get their way working. That is great! Thank you both for your efforts! But i think we should discuss the advantages and disadvantages of both solutions and try to find the way we want. From my point of view: There are many different modules and it is a major task to make single ports for them. This is an error prone but working way, as rubygems teached us. But this as the advantage that everything is clear and known Creating bundles saves as the port managing, which is a huge advantage. But we need a way to guarantee that everything is know before installing (no fetching of new libs after fetch-phase). Also how to you handle multiple usage of the same library? Multiple installation, references, etc? Currently it is not clear how do you solve it. Can you please explain this to me? (In reply to Torsten Zühlsdorff from comment #31) nodejs.mk fetches everything needed for the port during the fetch phase, then packages port with all dependencies so that this port always works the same way. Build doesn't require the network access. We use shrinkwrap feature that npm provides and recommends to use to freeze the dependencies. I just don't see how can individual ports be created for all dependencies. This is a very large amount of work to maintain them. Also please note that different projects can require different versions. How can having a single version on ports suit such requirement? Also if the older distfile disappears on the source, this will break the individual ports the same way. So besides saving the space having them doesn't provide any advantage in NodeJS case. I currently have 5 ports working, and looking at some other, larger projects to make ports for. Making ports for simple projects is trivial, finding larger projects that can also be built on FreeBSD seems to be a challenge (few of the ones that I found require large external C++ projects to work that weren't ported or require qt-5.5). Created attachment 164721 [details]
Updated patch
Made shrinkwrap deterministic.
Created attachment 164745 [details]
patch
Implemented auto-plist.
Will finish this once 9 is out. Bug #213947 is doing the same thing. Can you and Bradley combine your efforts? (In reply to Adam Weinberger from comment #36) I'm also working in Nodejs infrastructure, https://www.freebsd.org/news/status/report-2015-10-2015-12.html#Node.js-Modules Hi! This is blocking some other work. Are there any news on updated NodeJS infra? There were a few competing versions of a USES=node thing, so maybe Olivier could pitch in. (Also, the current code, here, is pretty unreadable, and completely undocumented) In the past I worked on NodeJS USES framework. (my repo [1] is outdated, I'm no longer interested to work on it, feel free to clone, update or whatever). You can find a Draft [2] of Porter's Handbook, and a complete list [3] of ports available. A NodeJS module (not extension, which are often written in C or C++) must contains package.json and one or several JavaScript files that's all. My USES framework tries to follow this guideline. Not mentioned in documentation page, but it supports DEFAULT_VERSION variable (available in /etc/make.conf). Module are downloaded from the NPM registry, and pkg-plist is build automatically. Users can override main targets (without to be in conflict with targets of framework). [1] https://app.assembla.com/spaces/cozycloud/subversion/source [2] https://people.freebsd.org/~olivierd/porters-handbook/uses-nodejs.html [3] https://people.freebsd.org/~olivierd/nodejs-ports.txt Is this ready to move forward? It's blocking a few other PRs. In its current state, it is overly complicated and quite unreadable, so, no, it is not ready to move forward. Moin moin We have a Mk/Uses/nodejs now with 46ce086c7130aedb8b349864afaf938489d070de So, I'll close this one. mfg Tobias |
Created attachment 163164 [details] patch This module is to support NodeJS ports, see dependencies. (ex. EtherPad, EtherCalc, etc) I am creating this bug report in order for the attached patch to be possibly used by those who wants to use EtherPad. However, there are few NodeJS bugs that need to be resolved until this can be committed. Without this such NodeJS-based ports don't pass poudriere build, and have to have NO_CHECKSUM=yes set, but still can be installed and used locally.