Created attachment 156160 [details] shar with new port www/rubygem-html-pipeline While working for the gitlab port, i figured out the need for this port. SHAR attached.
what is this "gitlab" port? I see a bunch of rubygem ports added. Are these all being added for some future port? And is it fair to say the ports are useless without it? For one you are adding a port without providing any description of what the port does, why it's need, etc -- just "here, commit this". I think all of these ports need real descriptions so people understand what's trying to be accomplished here. less text = more like to be ignored.
Hello John, the "gitlab" port is a future port of the GitLab-Software which is similar to GitHub. It's fair to say, that some of the added rubygem-ports have no value without the gitlab-port. Nearly every rubygem port with "gitlab" in its name is without value while the gitlab-port itself is missing. I get you're "to less information"-point. I falsely believed, that everything is clear, because i wrote it to the mailinglist and discuss the progress with xmj@, ruby@ and mmoll@. I'm sorry about that. In short: the future port "GitLab" has an awful long list of dependencies. At least it depends on 96 rubygems. There are missing gems and outdated gems. I've documented all of them (with links to PRs) here: http://ports.toco-domains.de/gitlab-gem-dependencies.html At the moment we are in a testing stage. I'm trying to put all the dependencies together to make the work at the gitlab-port possible. I have a copy of the ports-tree where i'm working on. Therefore it is possible to create "a big patch". But i didn't think this is useful. Many of the patches have there own value, especially the updates of existing ports.
it sounds like a nightmare. People are asking for rubygems to be removed in general since they can be installed from outside ports. Can't you package all 96 rubygems in a single port? So they can be installed / removed together? Who wants to commit ~100 ports just to get gitlab? (e.g. why is it vital to have gitlab in ports?) I'd recommend thinking about how to make this maintainable because right now it looks awful.
It's possible to install all rubygems only with the rubygem tool and completely without pkg. I've wrote about this topic with xmj@. I will quote him, because that's why i do the things this way: === Start === GitLab comes with an unmanageable amount of dependencies, which (as per ruby@ policy) may not be installed via bundle, and have to be (or become, if not yet available) ports themselves. Using ``bundle`` is taboo mostly because libraries thus installed don't register with pkg, and can't be deinstalled properly using that tool either. === End === For me this is very convincing. Also it comes with the advantage, that we can provide binary-packages for pkg, which currently works. The GitLab port itself would be great. More than 100.000 organizations uses this software. And it is relatively hard to install and maintained. The FreeBSD-Ports tree offers a very simple method for this (compared to others). And it allows freedom. There are just a few gitlab-packages in other distributions and most of them comes with a full software-stack you cannot change. Even the possibility to change the software-stack is enough for my company to sponsor some working-time at the project and considering the use of FreeBSD. Normally the use Ubuntu and nothing else.
(In reply to Torsten Zühlsdorff from comment #4) > It's possible to install all rubygems only with the rubygem tool > and completely without pkg. I wasn't advocating that. It is recognized that if the gem is needed, it needs to be installed with pkg. It was standalone gems there were targeted. I am saying, don't have 96 ports. Have one port with 96 gems, essentially creating your own bundle. It's not a standard thing to do, but it might be the best approach in this case -- if the vast majority of these gems only have a single purpose.
This could be a good approach. But from the 96 dependencies 80 were already in the ports before starting the work. So i don't think is practical to redo the existing work.
Hmm, it would be interesting to know about these 80: 1) are they there to support another port? (if so, which?) 2) the general reason they were there to begin with Yes, this information obviously changes things (bundle could go from 96 to 16) but it also could be an opportunity to contract rubygems. Remember there is a sentiment to get rid of them, so if some portion of the 80 existing ports can be moved to a new bundle port, it would probably be welcome. Extra work, sure. That's why I would be interested in knowing how each of these 80 came to be (obviously rubygems used by another port are not eligible for bundling)
You can see the other 80 ports also in this list: http://ports.toco-domains.de/gitlab-gem-dependencies.html There in the part, which is not red. As i can see there are libs which provides others. But i do not have a clue, why they are their. I wasn't invoked in them and i could not help you with this question.
Each port has a log history available on freshports. If I were going to try to figure out why a port existed, I'd look there, which probably references a PR which has even more history. Is is a lot of work, but it is certainly not an impossible or even unreasonable task, especially when rubygems that are dependencies of other ports can be removed off the list. maybe the list would shrink from 80 to 20, who knows?
John, the approach of packaging Rubygems in general, and especially Rails specific ones was always a controvese topic. Personally I always want everything OS packaged, that's why I'm putting energy into porting gems anyway. In regard to the general usefulness: except a few GitLab specific ones, it potentially helps other applications to get packaged also, especially newer versions of Jekyll or Redmine. In regard to a potential GitLab port: We're trying it the second time now, personally I think it would be better to target 7.11 (instead of 7.10 now) as everything is more tedious than expected. However, it will be a lot less work to do when all the dependent gems for 7.10 are getting in now.
Well, again, I never said gems shouldn't be packaged. I was suggested they be packaged as a bundle. I'm not very swayed about "potential". The idea is that anyone that needs them for extra-port things will install them some other way. If a gem has no identified ports-related reason to exist, it should be considered for removal. According to ehaupts USE_* counter, there are over 1000 rubygem ports. That's a bit insane. I've heard other people question this too, so maybe so kind of policy should be expressly established about rubygems because it seems to be getting out of control (although I recognize the same argument about perl ports can be made)
If this should be discussed further, please open a thread at ruby@. Just very few very short points from me: > People are asking for rubygems to be removed in general since they can be installed from outside ports. This point is also valid for a lot of Perl/Python ports with CPAN and pip. If we follow these people, we can just remove all ruby* ports and start using RVM and bundler/gem, like told by the internet and the people developing most gems, esp. on Mac OSX. > Can't you package all 96 rubygems in a single port? I don't think this would buy anything in regard to complexity (because that would have to get represented in bsd.ruby.mk and friends) and maintainability (because updates of single gems inside the bundle would need to get handled, too). Also, if a new port depends on a gem that's inside this bundle, it has to get extracted again. If we follow that approach, we can just remove all rubygem-* ports and most of the ruby-* ports, identify some applications, that should be ported and package these in an Omnibus (https://github.com/chef/omnibus) like way. > maybe so kind of policy should be expressly established about rubygems because it seems to be getting out of control (although I recognize the same argument about perl ports can be made) Let's say I wrote a script that uses gem A which depends on gem B and gem C and nothing else in the ports tree depends on these gems otherwise, I can now just install that set of ports or even create a local port of my script with the correct dependencies and all is set. I know people who use devel/rubygem-fog to control their VMs and I use devel/rubygem-kafo for some stuff that will never hit the internet, let alone the ports tree. I think it's wrong to just count dependency in the ports tree for Perl/Python/Ruby modules. We simply can not know in which ways these are used by people out there. Taking away all the ports for these modules would make my life harder and maybe require me to open some 3rd party repo with quite a bit of packages I would have to maintain alone. If we start to purge all these "unused" module ports, there will be an unknown (I can't say it's a huge one) number of users getting kicked in their teeth.
(In reply to Michael Moll from comment #12) > If this should be discussed further, please open a thread at ruby@. Well, I can't say "ruby@" has earned this based on their response to the staging issue. > If we follow these people, we can just remove all ruby* ports and start > using RVM and bundler/gem, like told by the internet and the people > developing most gems, esp. on Mac OSX. No, we are not including dependencies of ports here. And yes, I've already conceded the same sort of argument can be made for perl. > I don't think this would buy anything in regard to complexity (because that > would have to get represented in bsd.ruby.mk and friends) and > maintainability (because updates of single gems inside the bundle would need > to get handled, too). Also, if a new port depends on a gem that's inside > this bundle, it has to get extracted again. Why would they need to be updated? As long as gitlab builds and works. Yes, if a new port needs the same rubygem, then it could be split out separately. > If we follow that approach, we can just remove all rubygem-* ports and most of the ruby > * ports, identify some applications, that should be ported and package these in an > Omnibus (https://github.com/chef/omnibus) like way. maybe. > Let's say I wrote a script that uses gem A which depends on gem B and gem C and nothing > else in the ports tree depends on these gems otherwise, I can now just install that set > of ports or even create a local port of my script with the correct dependencies and all > is set. I know people who use devel/rubygem-fog to control their VMs and I use > devel/rubygem-kafo for some stuff that will never hit the internet, let alone the ports > tree. So what's stopping you from installing the gem via normal way of getting gems? This was the premise -- that rubygems are installed normally another way, so rubygems in ports should be mapped to what depends on them and all extras ones removed. > I think it's wrong to just count dependency in the ports tree for Perl/Python/Ruby > modules. We simply can not know in which ways these are used by people out there. Taking > away all the ports for these modules would make my life harder and maybe require me to > open some 3rd party repo with quite a bit of packages I would have to maintain alone. same statement. In the case of orphaned ports, we aren't taking anything away. They are still available during normal mechanism. > If we start to purge all these "unused" module ports, there will be an unknown > (I can't say it's a huge one) number of users getting kicked in their teeth. You know about DEPRECATED and EXPIRATION_DATE right? You just set it with a date far out. If somebody is affected they'll let us know before it gets deleted. Again, unless I've been misinformed, users get easily get any rubygem through another method and in fact, that other method is the normal way of getting them. Have I been misinformed?
You have not. As said, same for all perl and python modules...
I am under the impression getting perl modules is significantly more difficult (and not necessarily the normal way). I know nothing about python. That said, perl has the same criticisms. Should the ports tree just be a copy of CPAN?
A commit references this bug: Author: mmoll Date: Sun May 10 23:38:54 UTC 2015 New revision: 386035 URL: https://svnweb.freebsd.org/changeset/ports/386035 Log: new port: textproc/rubygem-html-pipeline GitHub HTML processing filters and utilities WWW: https://github.com/jch/html-pipeline PR: 199820 Differential Revision: https://reviews.freebsd.org/D2508 Submitted by: Torsten Zuehlsdorff <ports@toco-domains.de> Approved by: swills (mentor) Changes: head/textproc/Makefile head/textproc/rubygem-html-pipeline/ head/textproc/rubygem-html-pipeline/Makefile head/textproc/rubygem-html-pipeline/distinfo head/textproc/rubygem-html-pipeline/pkg-descr
John, you're not misinformed about the "normal" way to get ruby-gems. But you misses some practical points. For example i'm only able at 4 out of my 11 servers to install the rubygems the "normal" way. The other 7 servers are IPv6 only and rubygem.ort does not serve the gems via IPv6. That is ridiculous, especially after a number of requests for IPv6 support. That is no problem when using pkg. I create my own repositories and could serve rubygems as pkg without problems. Also it is very helpful to have the ability to use *one* method - the pkg - to distribute software-stacks. It is very easy to write some programs, which manage different server roles and manage the software from a central point. To do this and support others, becomes very hard and error prone. I would need support for pkg, rubygems, perl, python, npm, (PHP) composer and some more. I do not even think at typical addons for some software - hey, there are also some Firefox and Thunderbird plugins in the portstree. Or modules for other software. Rubygems as ports do have disadvantages. There could be better solutions and while working at the gitlab port, i become a better understanding about the infrastructure, advantages and disadvantages. I already have some ideas for better rubygem integration which will satisfy both sides. :) But i believe a PR is a bad place to discuss this issue. Lets move to another place for this.
@Michael: Thanks for committing! :) The status of this PR needs a change! :)
(In reply to Torsten Zühlsdorff from comment #17) Torsten, Can your reply be summarized as, "Due to serious flaws in the rubygem architecture, FreeBSD ports is compelled to duplicate these repositories"? And then there is the philosophical discussion which is not limited to rubygems: Should we maintain ports that nothing depends on just in case somebody needs it for non-ports work? My feeling on the second part is "no". It's very easy to measure when a port is not a dependency of another port. It would be very easy to have a standard policy that rubygems that aren't directly in place to support another port could be removed. on the other hand it is basically impossible to understand if somebody is using a port for their own reasons so it would be impossible to remove the port, ever, because it *might* be used. See the point? P.S. how come nobody that is affected by IPv6 doesn't just mirror the repo as a service to the other IPv6 users?
John - all ports philosophical arguments aside. I've heard from many of my own clients that they'd love to have GitLab in ports, so that there's an easy, officially supported, way to install it. I've spoken with the GitLab people about it when porting all those named 80 dependencies last year, and they appreciated the fact that FreeBSD people were in a position to take care of it. Now we have a team of people contributing the work (thanks Torsten and Michael), and a team of people making sure everything is in line with policies set by the ruby@ team. Yes all this work might be rendered moot in the future once we get integration for pkg with third party repos, including gems. And all timelines I've heard about that happening have fallen flat. So let's stop the bikeshed, get GitLab into the tree, make and keep actual FreeBSD users (and contributing businesses) happy.
I take exception to you referring to my comments as a bikeshed. I find it insulting.
(In reply to John Marino from comment #19) > Can your reply be summarized as, "Due to serious flaws in the rubygem > architecture, FreeBSD ports is compelled to duplicate these repositories"? John, english is not my foreign language, so i would say you're summarize is a little bit to hard in nuance, but it is okay. But i'm not sure. Also the FreeBSD ports offers (often more than one) stringent way of administration. Handling different repositories makes this advantage disappear, because the different systems bring their own - often not so good - ways of administration. > And then there is the philosophical discussion which is not limited to > rubygems: Should we maintain ports that nothing depends on just in case > somebody needs it for non-ports work? > My feeling on the second part is "no". It's very easy to measure when a port > is not a dependency of another port. It would be very easy to have a > standard policy that rubygems that aren't directly in place to support > another port could be removed. on the other hand it is basically impossible > to understand if somebody is using a port for their own reasons so it would > be impossible to remove the port, ever, because it *might* be used. I know that there are many ports which are very less in use. I maintain some of them. But i feel that it is a hard philosophical topic. Sometimes ports have just value to very little number of persons - but for them it is crucial. As long as the amount of work is in the correct relation that is fine for me. And the Gitlab port is a often wanted port. It is also in the list of wanted ports: https://wiki.freebsd.org/WantedPorts (By the way: could someone add me to "Who is working" section?) > P.S. how come nobody that is affected by IPv6 doesn't just mirror the repo > as a service to the other IPv6 users? As always: there are much more complains out in the world than hitting a person involved in the related project(s). I would volunteer to setup a mirror. Last year i contacted a bigger amount of projects and offered help to get IPv6 set up or a mirror. Most projects didn't know about the topic or its importance. And persons hit by the problem just fetch the files another way, because its easier then writing many mails and helping.
Torsten, My "summary" was a bit sarcastic. Basically I was pointing out that the justification for many of the rubygem ports traced back to flaws in the distribution of them. I wouldn't mind "crucial" ports staying as long as one person that finds them crucial is listed as the maintainer. If it's just maintained by "ruby@" then how can anyone know it is still needed? As far as the mirror goes, I would think you could just set one up (perhaps with the help of the original site) and after the link is published people will find it and use it. I didn't understand why feedback from potential users is even needed. Just mirror the entire site and have it sync daily.
closing this for now, as it was committed. As a last point from me, I'd highly welcome going away from persons as maintainers, but I saw especially in the last days that the way to work I'm used to from other, more collaboratively maintained projects is often not applicable here.