Bug 255980

Summary: net-mgmt/netdata: Update to 1.31.0
Product: Ports & Packages Reporter: Namkhai B. <nk>
Component: Individual Port(s)Assignee: Matthias Andree <mandree>
Status: Closed FIXED    
Severity: Affects Only Me CC: diizzy, mandree
Priority: ---    
Version: Latest   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
net-mgmt/netdata: Update to 1.30.1
none
net-mgmt/netdata: Update to 1.31.0
none
Patch for netdata (WIP)
none
Patch for netdata (WIP) v2
none
net-mgmt/netdata: Update to 1.31.0 with customized dashboard
none
Patch for netdata none

Description Namkhai B. 2021-05-18 19:17:22 UTC
Created attachment 225073 [details]
net-mgmt/netdata: Update to 1.30.1

This patch updates net-mgmt/netdata to version 1.30.1.

Tested with https://docs.freebsd.org/en/books/porters-handbook/quick-porting/#porting-testing, but haven't run portlint as I think the changes aren't big enough.

I tested this on my production server, and everything is working fine (although I haven't enabled many plugins yet).

I also want to apply as a maintainer for this port, but I didn't add yet my email to MANTAINER as I don't know what the protocol for this is.
Comment 1 Daniel Engberg freebsd_committer freebsd_triage 2021-05-19 00:31:14 UTC
Hi,

1.30.X will fail to build if you don't have GCC installed (hardcoded). This issue is already fixed upstream but there are more issues with the FreeBSD plugin that aren't fixed in this release. We're waiting on https://github.com/netdata/netdata/pull/11159 too so we might as well hold off until it's released which should be soon.

Best regards,
Daniel
Comment 2 Namkhai B. 2021-05-19 00:47:21 UTC
Weird... I built the package on a Bastille jail with no GCC installed:

root@packages:~ # pkg info | grep gcc
root@packages:~ #

But I do agree we should wait for the next release, I hadn't looked into the PRs or issues.
Comment 3 Namkhai B. 2021-05-19 17:04:47 UTC
Created attachment 225088 [details]
net-mgmt/netdata: Update to 1.31.0

This patch updates net-mgmt/netdata to version 1.31.0
Comment 4 Namkhai B. 2021-05-19 17:08:30 UTC
1.31.0 was just released today!

There were some breaking changes in the dashboard, and I'm not entirely sure I made the correct choices.

Specifically in files/extra-patch-web_gui_index.html and files/patch-web_gui_index.html, as they removed web/gui/index.html, and the only thing I found that could match what the patches do is web/gui/old/index.html.
Comment 5 Daniel Engberg freebsd_committer freebsd_triage 2021-05-19 21:10:58 UTC
Hi,

We still need https://github.com/netdata/netdata/pull/11159, preferaly remove the rather distracting social networks window and the update reminder stuff. Unfortunately it seems be a bit of a hassle without rebuilding the dashboard. :/
Comment 6 Namkhai B. 2021-05-20 01:45:34 UTC
All right, we'll wait for the next version (released with pr#11159 merged) gets out.

In the meantime, I'll check what I can do to build a customized dashboard. I'm pretty busy until the end of this week, tho, so it'll have to wait until next week.
Comment 7 Daniel Engberg freebsd_committer freebsd_triage 2021-05-20 08:15:35 UTC
Cloud functionality is also broken, I personally don't care about it so in worst case we could mark it a BROKEN for now. Issue submitted upstream as https://github.com/netdata/netdata/issues/11171
Comment 8 Daniel Engberg freebsd_committer freebsd_triage 2021-05-20 15:57:32 UTC
Created attachment 225129 [details]
Patch for netdata (WIP)

Update netdata to 1.31.0
Pull in PR #11159 - Thanks to mikael@ for troubleshooting and submitting a patch upstream
Define LICENSE_FILE
Backport upstream commits for mqtt_websockets to fix cloud functionality
Fix version reporting in netdata (cosmetic fix)
Rearrange Makefile slight for better readability
Comment 9 Daniel Engberg freebsd_committer freebsd_triage 2021-05-20 15:59:52 UTC
So what I would like to "fix" is to wait for upstream to merge the PR so we can refer to is as a commit and preferably "fix" (not a showstopper) patch the minified javascript code.
Comment 10 Daniel Engberg freebsd_committer freebsd_triage 2021-05-24 23:22:21 UTC
Created attachment 225238 [details]
Patch for netdata (WIP) v2

Backport commit rather than pull request from upstream
Comment 11 Daniel Engberg freebsd_committer freebsd_triage 2021-05-31 22:27:07 UTC
For the sake of documentation:

https://github.com/netdata/netdata/issues/11213 - Issue reported to make netdata not tell people to work around their package repo (if used)

https://github.com/netdata/netdata/issues/11214 - Using git and a git repo to determine version fails due to packaging

Since I don't think netdata will remove their links to "social networks" voluntarily I didn't file that as an issue upstream. It's not the end of the world if these aren't removed (imho) as I mentioned earlier.
Comment 12 Daniel Engberg freebsd_committer freebsd_triage 2021-06-05 09:13:21 UTC
@ Namkhai
Did you manage to figure out the minified js stuff otherwise I think we can commit it for now as is.
Comment 13 Namkhai B. 2021-06-05 21:50:30 UTC
Unfortunately I've been very busy this week with my work (outside of computers ;]). So I haven't been able to check what I can do.

The way I think this can work is if I build a custom dashboard on my desktop, and then use that as a diff for the package. I would either document the steps, or provide a script to update that on every netdata update.
Comment 14 Daniel Engberg freebsd_committer freebsd_triage 2021-06-10 07:12:52 UTC
Hi Namkhai,

That sounds great! Would you mind if we committed netdata as-is for now as it fixes quite a few bugs?

Thanks for looking into this!

Best regards,
Daniel
Comment 15 Namkhai B. 2021-06-10 17:15:52 UTC
Yeah, I'm OK with it.

I'm currently stuck at the patching process of that dashboard:

I got a patch, it's living in a GitHub fork (because I don't want to pollute the ports tree with a 20MB+ patch), but I can't get it to apply cleanly, as there are some file renames, and BSD patch isn't handling them correctly.

Using GNU patch works, but I don't know how we could tell FreeBSD's make to use gpatch instead of patch in the patch process.

(lots of `patch`es :D)

The patch I currently have is over here: https://paste.c-net.org/BradysCollect (based on `Patch for netdata (WIP) v2`)
Comment 16 Daniel Engberg freebsd_committer freebsd_triage 2021-06-11 23:25:49 UTC
Namkhai,

I very much appreciate your efforts as it not something I would be able to figure out on my own. What makes it such a large patch and do you have link to that repo?

I'm a bit concerned about patch size for the js file(s) but I guess that's not the end of the world unless you have a better approach? I would expect a patch to break between release though due to minification though.
Comment 17 Namkhai B. 2021-06-12 00:56:08 UTC
So, here's the process I followed to build the custom dashboard:

1. I have a fork at https://github.com/forkbomb9/dashboard, in which the branch `freebsd` removes the social media components.

2. I then build the dashboard (using Node LTS), and pack `build/` in `dashboard.tar.gz`.
This file can be found here: https://github.com/forkbomb9/dashboard/releases/tag/v2.17.1-bsd0.

I then use that as a basis to update the dashboard in netdata:

1. go to web/gui, and in bundle_dashboard.py, change URLTEMPLATE to point to my fork, e.g. instead of github.com/netdata/dashboard use github.com/forkbomb9/dashboard.

2. call bundle_dashboard.py with my custom release tag :./bundle_dashboard.py v2.17.1-bsd0.

This is now the customized dashboard. This procedure is only to generate the patch, it has to be done on every dashboard update (it's not that hard).

Then, to generate the actual patch, I commit the changes to my `netdata` fork in the `freebsd` branch. And then I use that commit as a patch in the Makefile (https://github.com/forkbomb9/netdata/commit/69bb76a794d8ae7fde1d9a1a8202fef3ec04726a)

This is the most efficient way I found to update the dashboard, while following the build procedure. The only thing that I have to do is update my `netdata` branch every time there's a package update, and it isn't very hard to do.

Another option would be to run bundle_dashboard.py on the build process, thus skipping the patch download. But that would pull Python as a build dependency...

I could also try to rewrite it in Lua, but that feels much hackier than the procedure I described above. And I don't have the skills in Bash/Sh to do it...
Comment 18 Namkhai B. 2021-06-12 18:56:27 UTC
All right, here we go!

Dashboard with things removed (image): https://paste.c-net.org/PitchersRoutes

Dashboard source code & patch: https://github.com/forkbomb9/dashboard/tree/freebsd
Dashboard build: https://github.com/forkbomb9/dashboard/releases/tag/v2.17.1-bsd1
Netdata patch (update to custom dashboard v2.17.1-bsd1): https://github.com/forkbomb9/netdata/commit/fd15080f65fde1c484a06cbc5a13a2853639a960
Branch here: https://github.com/forkbomb9/netdata/tree/freebsd

I'll upload shortly the patch. It's based on `Patch for netdata (WIP) v2`.

The route I went was to apply the patch manually with `gpatch`, and add the Git patch as a DISTFILE.

This afternoon I'll create a cleaner gist with the steps involved in updating the dashboard.
Comment 19 Namkhai B. 2021-06-12 18:57:45 UTC
Created attachment 225759 [details]
net-mgmt/netdata: Update to 1.31.0 with customized dashboard
Comment 20 Namkhai B. 2021-06-12 19:00:20 UTC
Final note: Now it's very easy for me to remove anything else.

Here's a .txz package if you want to test it: https://paste.c-net.org/BackupAdmiring
Comment 21 Daniel Engberg freebsd_committer freebsd_triage 2021-06-13 05:31:07 UTC
Created attachment 225763 [details]
Patch for netdata

Some minor fixes that portlint complained about
Restore original dashboard for cloud users
Comment 22 Daniel Engberg freebsd_committer freebsd_triage 2021-06-13 05:32:06 UTC
Fantastic work! Thanks for the effort you've put into this!
I did some minor fixups of the makefile that portlint complained about and while not ideal (I personally don't really care) we need to keep the login part at least for the cloud functionality so users wanting to use it "needs" to have the original dashboard. There's a little bit of code duplication as the ports framework didn't like if statments within do-patch section but I think we can live with that.

I've also tested this patch in poudriere so this should be the final patch for this version.
Comment 23 Namkhai B. 2021-06-14 16:48:37 UTC
Daniel, nice!

I hadn't thought about this breaking cloud functionality for some users, good catch!

I think too this is the final version.
Comment 24 commit-hook freebsd_committer freebsd_triage 2021-06-14 21:00:04 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=40501c149e07527bd1d86292921350bde644ef0b

commit 40501c149e07527bd1d86292921350bde644ef0b
Author:     Namkhai B <namkhai.n3@protonmail.com>
AuthorDate: 2021-06-12 18:57:45 +0000
Commit:     Matthias Andree <mandree@FreeBSD.org>
CommitDate: 2021-06-14 20:58:25 +0000

    net-mgmt/netdata: update to 1.31.0 with...

    customized dashboard.

    Some cleanups by Daniel Engberg <daniel.engberg.lists@pyret.net>.

    Changelog:      https://github.com/netdata/netdata/releases

    PR:             255980

 net-mgmt/netdata/Makefile                          |  54 +++++++---
 net-mgmt/netdata/distinfo                          |  10 +-
 .../files/extra-patch-web_gui_index.html (gone)    |  28 -----
 .../files/extra-patch-web_gui_main.js (gone)       |  15 ---
 ...ch-bump-mqtt_websockets-to-commit-6a4fba1 (new) | 112 +++++++++++++++++++
 net-mgmt/netdata/files/patch-configure.ac          |  20 +++-
 .../netdata/files/patch-web_gui_index.html (gone)  |  17 ---
 net-mgmt/netdata/pkg-plist                         | 119 ++++++++++++++++++++-
 8 files changed, 294 insertions(+), 81 deletions(-)