Bug 204128 - [NEW PORT] devel/artifactory: Universal Artifact Repository Manager
Summary: [NEW PORT] devel/artifactory: Universal Artifact Repository Manager
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Thomas Zander
URL:
Keywords: feature, patch
Depends on:
Blocks:
 
Reported: 2015-10-29 15:46 UTC by David Harrigan
Modified: 2015-11-09 20:41 UTC (History)
2 users (show)

See Also:


Attachments
SHAR file. (71.02 KB, text/plain)
2015-10-29 15:46 UTC, David Harrigan
no flags Details
UIDs/GIDs diff. (922 bytes, patch)
2015-10-29 15:48 UTC, David Harrigan
dharrigan: maintainer-approval+
Details | Diff
SHAR File (10.12 KB, text/plain)
2015-10-29 15:55 UTC, David Harrigan
no flags Details
SHAR File (10.36 KB, text/plain)
2015-10-30 09:31 UTC, David Harrigan
dharrigan: maintainer-approval+
Details
SHAR file (10.32 KB, text/plain)
2015-11-02 17:01 UTC, David Harrigan
dharrigan: maintainer-approval+
Details
SHAR File (10.32 KB, application/x-shar)
2015-11-07 08:23 UTC, David Harrigan
dharrigan: maintainer-approval+
Details
Content diff of %%APP_HOME%% before and after starting artifactory for the first time (22.67 KB, text/plain)
2015-11-07 11:13 UTC, Thomas Zander
no flags Details
SHAR File (10.32 KB, application/x-shar)
2015-11-09 12:36 UTC, David Harrigan
dharrigan: maintainer-approval+
Details
SHAR File (10.32 KB, application/x-shar)
2015-11-09 13:53 UTC, David Harrigan
dharrigan: maintainer-approval+
Details
UID/GID for Artifactory (923 bytes, patch)
2015-11-09 14:46 UTC, David Harrigan
dharrigan: maintainer-approval+
Details | Diff
SHAR File (12.68 KB, text/plain)
2015-11-09 17:21 UTC, David Harrigan
dharrigan: maintainer-approval+
Details
Proposed update for artifactory shar (untested!) (10.72 KB, text/plain)
2015-11-09 19:26 UTC, Thomas Zander
no flags Details
SHAR File (10.73 KB, text/plain)
2015-11-09 19:50 UTC, David Harrigan
dharrigan: maintainer-approval+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description David Harrigan 2015-10-29 15:46:22 UTC
Created attachment 162568 [details]
SHAR file.

Hi,

I've put together a new port for Artifactory (described here: https://www.jfrog.com/open-source). It's a Java application that sits between you and the outside world, and if properly configured, caches artifacts locally (such as JAR files, Zip files, npm packages and so on), so that the next time they are "fetched", they are retrieved locally rather than fetched from the remote server. This saves time and bandwidth and is a great boon for development.

-=david=-
Comment 1 David Harrigan 2015-10-29 15:48:24 UTC
Created attachment 162569 [details]
UIDs/GIDs diff.
Comment 2 David Harrigan 2015-10-29 15:55:26 UTC
Created attachment 162570 [details]
SHAR File

Removed the .git directory.
Comment 3 David Harrigan 2015-10-30 09:31:36 UTC
Created attachment 162593 [details]
SHAR File

Updated with a more descriptive pkg-message.in file.
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2015-10-30 09:36:33 UTC
Nice work David!

If and when you're happy with the version of the attachment that has been submitted, please set the maintainer-approval flag to (+) on those attachments that you would like a committer to look at.

Please also confirm that this passes QA [1], and if not, obsolete the existing attachments after updating them.

[1] https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/testing.html
Comment 5 David Harrigan 2015-10-30 10:21:28 UTC
Hi,

Thank you. I've went through the testing section again on the Porters Handbook and also tested the port using Poudiere. All appears to be well. A couple of WARNs on Portlint, but I hope not show-stoppers.

$ portlint -a
WARN: Makefile: [52]: possible use of "${CHMOD}" found. Use @(owner,group,mode) syntax or @owner/@group operators in pkg-plist instead.
WARN: Makefile: no ftp/http mirror in MASTER_SITES for users behind a proxy.
0 fatal errors and 2 warnings found.

-=david=-
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2015-10-30 10:27:22 UTC
Thanks for the feedback David :)
Comment 7 Brad Davis freebsd_committer freebsd_triage 2015-11-02 14:30:44 UTC
Why not remove the ${CHMOD} and use @mode in the pkg-plist?
Comment 8 David Harrigan 2015-11-02 17:00:55 UTC
Hi Brad,

Thank you for your feedback. I've updated the pkg-plist to use @mode and removed the ${CHMOD} from the Makefile. Please retest.

Thank you.

-=david=-
Comment 9 David Harrigan 2015-11-02 17:01:14 UTC
Created attachment 162713 [details]
SHAR file
Comment 10 Thomas Zander freebsd_committer freebsd_triage 2015-11-07 08:09:42 UTC
Can we use the most recent 4.2.2 version instead?
Comment 11 David Harrigan 2015-11-07 08:11:36 UTC
Hi!

I don't see why not :-) It was just released. I'll update the port, do a test locally, then push up a new SHAR file.

Thank you kindly for looking :-)

-=david=-
Comment 12 David Harrigan 2015-11-07 08:23:19 UTC
Created attachment 162873 [details]
SHAR File

Update to Artifactory 4.2.2.

-=david=-
Comment 13 Thomas Zander freebsd_committer freebsd_triage 2015-11-07 08:40:58 UTC
(In reply to David Harrigan from comment #12)

Thanks for the quick response. Always happy to use the latest and hopefully greatest version.

Let's quickly have a look at pkg-plist. Why do all the files and dirs belong to artifactory:artifactory? Normally all file and dir permissions should be default with only dirs and files that the artifactory processes need to write having different permissions (log files, status files, etc.).
Comment 14 David Harrigan 2015-11-07 08:46:16 UTC
Hi,

I found that when I booted Artifactory (when the /usr/local/artifactory directory was owned by root), it failed since the Artifactory user could not write into the directory and thus create all the files/directories it requires. 

I suppose I could have made the directory group writeable and put the group ownership of the directory to "artifactory". I believe, given that inside the Artifactory directory there are only a few directories and files it was simply easier to let them all belong to Artifactory. However, I'm happy to play around with a few other ways of doing this based on what you can suggest? :-)

-=david=-
Comment 15 Thomas Zander freebsd_committer freebsd_triage 2015-11-07 10:57:55 UTC
(In reply to David Harrigan from comment #14)

In general you should always try to use the @marcos as sparingly as possible. Granted, sometimes they are needed, and sometimes apps will refuse to run properly if they can't have it their way, and in those cases we should apply them to make it work.
But it should be really really really not necessary for something like
%%APP_HOME%%/COPYING
to be owned and writable by the application. Similar goes for %%APP_HOME%%/etc/artifactory.config.xml.sample. Why should the application modify this file ever?

For this port, pkg-plist is fairly small, it should be relatively quick to find the minimum set of dirs and files that require non-default owner, group or modes.

I don't know whether you are using poudriere, but you should. It helps a lot for issues like this.
Comment 16 Thomas Zander freebsd_committer freebsd_triage 2015-11-07 11:13:20 UTC
Created attachment 162875 [details]
Content diff of %%APP_HOME%% before and after starting artifactory for the first time
Comment 17 Thomas Zander freebsd_committer freebsd_triage 2015-11-07 11:38:05 UTC
(In reply to Thomas Zander from comment #16)

Have a look at attachment 162875 [details] . I made a diff of the %%APP_HOME%% dirs and subdirs before and after starting artifactory using the default config for the first time.
It creates lots of files in data, logs and tomcat/webapps and changes etc. All those files would be leftovers in LOCALBASE. This should not happen.

I would split this into two parts:
1) The 'static' part of artifactory (i.e. files that we only need to read and execute, but never touch or write) should go to what is currently APP_HOME in your port. Nothing there should change after installation, such that it can be cleared perfectly by pkg delete.
2) The 'dynamic part' (i.e. logs, pidfiles, tomcat webapps and so on, everything that changes at runtime) should go to /var. pidfile into /var/run, the rest into an artifactory-owned subdir in /var. Those dirs must be registered with the @dir macro in plist. Since this changes at runtime, there will be leftover files after deinstall. There must be a pkg uninstall message informing the user which dirs may need manual cleanup.

Could you take a look?
Comment 18 David Harrigan 2015-11-07 17:48:59 UTC
Hi,

This is great feedback. I'll start looking at the issues and see what I can do to move this port on to the stage where it is ready!

Thank you very much :-)

-=david=-
Comment 19 David Harrigan 2015-11-09 09:22:12 UTC
Hi,

Question:

Should "var/artifactory" be "/var/artifactory" or "/usr/local/var/artifactory" (and same question for var/run?)?

Thank you.

-=david=-
Comment 20 David Harrigan 2015-11-09 12:36:26 UTC
Created attachment 162920 [details]
SHAR File

Hi,

Based upon recent feedback, here is an improved installation of Artifactory. I've tried to keep things separate, in their own /var/artifactory directory where appropriate. 

Artifactory does create some temporary files in /usr/local/artifactory (tomcat compiled files), but where done, I've clearly indicated in a deinstallation message which directories to clean up.

Please review and let me know if I need to improve upon this further :-)

Thank you.

-=david=-
Comment 21 David Harrigan 2015-11-09 13:53:25 UTC
Created attachment 162922 [details]
SHAR File

Further improvements. Hopefully good to go now! :-)

-=david=-
Comment 22 David Harrigan 2015-11-09 13:54:39 UTC
Please note, the new attachment, SHAR file, correctly creates tomcat 'var' temporary directories, thus allowing a clean removal of Artifactory.
Comment 23 David Harrigan 2015-11-09 14:46:26 UTC
Created attachment 162923 [details]
UID/GID for Artifactory

Brings this up-to-date with latest changes made elsewhere to UIDs/GIDs.

-=david=-
Comment 24 Thomas Zander freebsd_committer freebsd_triage 2015-11-09 17:18:26 UTC
(In reply to David Harrigan from comment #22)

Are you sure you uploaded the right attachments? Just a superficial look:
- PORTVERSION is 4.2.0 instead of 4.2.2
- owner macros in plist are still never reset
- PID file goes into ${APP_HOME}/bin/${PORTNAME}.pid
- APP_HOME expands to full path in plist expansion (which it should only do outside PREFIX)
- superfluous @dir entries in plist
- The shar does not contain anything with 'var' at all (except rcvar= ...)
- the gid/uid diff does not patch correctly to the latest changes

Could you please double check?
Comment 25 David Harrigan 2015-11-09 17:21:56 UTC
Created attachment 162928 [details]
SHAR File

Hi,

Terribly sorry for that! I forgot to redirect the contents of shar into a file, so when I generated the shar file, it was going to console...then I was uploading the old shar file!!!

d'oh!

-=david=-
Comment 26 Thomas Zander freebsd_committer freebsd_triage 2015-11-09 19:26:50 UTC
Created attachment 162936 [details]
Proposed update for artifactory shar (untested!)

I have made a few proposed changes, please review:
- Remove superfluous (I believe) files
- Reduce, simplify plist
- Use variable substitution for several variables that otherwise would have to be manually sync'ed over multiple files
- If I am not mistaken, there are multiple components of different LICENSE. Adapt LICENSE* accordingly
- Use lowercase spelling for shorter loop-variable, so it is clearer what the invariants and loop variables are

Please compare this with your most recent submission to see the differences.

Note that I have not *tested* whether the service actually *runs* flawlessly. You might want to double-check running it and tweak the shar until you are satisfied.
Comment 27 David Harrigan 2015-11-09 19:50:05 UTC
Created attachment 162937 [details]
SHAR File

Hi,

Attached is the updated SHAR file. It looks a lot better now thank you Thomas - just one thing, there was an error - it depends upon artifactory.default which you had removed in the updated Makefile. A quick test on Poudriere discovered that.

I've readded artifactory.default back in, tested and it appears to boot up successfully :-)

-=david=-
Comment 28 commit-hook freebsd_committer freebsd_triage 2015-11-09 20:32:14 UTC
A commit references this bug:

Author: riggs
Date: Mon Nov  9 20:31:42 UTC 2015
New revision: 401165
URL: https://svnweb.freebsd.org/changeset/ports/401165

Log:
  Add new port: devel/artifactory, a universal artifact repository manager

  PR:		204128
  Submitted by:	dharrigan@gmail.com (maintainer)

Changes:
  head/GIDs
  head/UIDs
  head/devel/Makefile
  head/devel/artifactory/
  head/devel/artifactory/Makefile
  head/devel/artifactory/distinfo
  head/devel/artifactory/files/
  head/devel/artifactory/files/artifactory.in
  head/devel/artifactory/files/pkg-message.in
  head/devel/artifactory/pkg-descr
  head/devel/artifactory/pkg-plist
Comment 29 Thomas Zander freebsd_committer freebsd_triage 2015-11-09 20:38:37 UTC
Here you go! Committed, thanks!
Have fun maintaining! :)
Comment 30 David Harrigan 2015-11-09 20:41:35 UTC
Thank you! 

I look forward to the challenges that brings :-)

Thank you kindly for your help and support in getting this out. I appreciate it!

-=david=-