Bug 207715 - [NEW PORT] archivers/py-borgbackup: Deduplicating backup program
Summary: [NEW PORT] archivers/py-borgbackup: Deduplicating backup program
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Raphael Kubo da Costa
URL:
Keywords: feature, patch-ready
Depends on: 203759
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-04 21:57 UTC by jjuanino
Modified: 2016-04-22 18:05 UTC (History)
5 users (show)

See Also:
rakuco: maintainer-feedback+


Attachments
shar file (2.43 KB, text/plain)
2016-03-04 21:57 UTC, jjuanino
rakuco: maintainer-approval+
Details
New makefile version (2.29 KB, text/plain)
2016-03-05 12:15 UTC, jjuanino
no flags Details
New makefile version (2.54 KB, text/plain)
2016-03-05 21:58 UTC, jjuanino
no flags Details
1.0.0 has been release a few hours ago (2.53 KB, patch)
2016-03-05 22:52 UTC, jjuanino
rakuco: maintainer-approval+
Details | Diff
1.0.1 has been release. I attach the new shar file (2.65 KB, text/plain)
2016-04-16 09:02 UTC, jjuanino
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description jjuanino 2016-03-04 21:57:17 UTC
Created attachment 167731 [details]
shar file

Dear FreeBSD committer,

I submit a port for making deduplicating backups (as attic or zbackup). Thanks.
Comment 1 Raphael Kubo da Costa freebsd_committer 2016-03-05 11:39:25 UTC
Hi,

Don't you need to set PKGNAMEPREFIX=${PYTHON_PKGNAMEPREFIX} like other Python packages do?

Additionally:
- Instead of setting MASTER_SITES like you did, you can do MASTER_SITES=CHEESESHOP instead.
- Can you clarify why you added that PACKAGE_BUILDING check?
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2016-03-05 12:08:57 UTC
The PACKAGE_BUILDING check looks like a copy of what was added to www/py-aiohttps in r388656 [1], that I just removed [2] due to it being a 3.x+ port, and the package builders only building for 2.7 (the DEFAULT_VERSION).

What instead needs to happen is that this ports dependencies need to have py3-* versions created for them (very very very unfortunately).



Other review items:

* Missing PKGNAMEPREFIX=PYTHON_PKGNAMEPREFIX
* PORTVERSION needs to be DISTVERSION for this version string
* post-install STRIP_CMD can use *.so instead of a for loop and should use PYTHONPREFIX_SITELIBDIR
* Upstream says the minimum Python versions is 3.4 [3]
* Use ${PYTHON_PKGNAMEPREFIX}name>=x.y for RUN_DEPENDS instead of on the .so files [4]
* Missing BUILD_DEPENDS on setuptools_scm [5]
* I see no "requires" reference to pylibacl. If this is *actually* needed, send a patch upstream

[1] https://svnweb.freebsd.org/changeset/ports/388656
[2] https://svnweb.freebsd.org/changeset/ports/410169
[3] https://github.com/borgbackup/borg/blob/master/setup.py#L10
[4] install_requires = ['msgpack-python>=0.4.6', ]
[5] setup_requires=['setuptools_scm>=1.7'],
Comment 3 jjuanino 2016-03-05 12:15:52 UTC
Created attachment 167738 [details]
New makefile version
Comment 4 jjuanino 2016-03-05 12:19:54 UTC
Hi,

> Don't you need to set PKGNAMEPREFIX=${PYTHON_PKGNAMEPREFIX} like other Python packages do?

Yes, you are right, it is needed.

> - Instead of setting MASTER_SITES like you did, you can do MASTER_SITES=CHEESESHOP instead.

Done



> Can you clarify why you added that PACKAGE_BUILDING check?

I am mimicking the archivers/attic port, but it seems that check is not neccesary... please let me know. The idea behind of this check is "do not build this package if the default python version (2.7) is not the same as the required by this port (3.3+)". But I can see a lot of 3.4 python ports in the pkg infraestructure. I am misunderstanding something, sure.

I attach a new shar file with the new suggestions. Thanks for your efforts, and excuse my poor english.
Comment 5 Raphael Kubo da Costa freebsd_committer 2016-03-05 12:29:19 UTC
koobs, can you check if this looks OK now?
Comment 6 jjuanino 2016-03-05 13:06:23 UTC
Hi, I will send you a new shar file tonight, trying to address the suggestions of your last response.

By the meanwhile, it would be nice if someone take a look to Bug 203759, as borg has a "soft" dependency with llfuse python. I would like to enforce a "hard" dependency between borg and llfuse. Thanks.
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2016-03-05 13:40:39 UTC
Pending new shar, both existing attachments should be obsoleted with that update:

Attachment -> Details -> Edit Details -> [x] Obsolete
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2016-03-05 13:41:43 UTC
@Raphael Once an updated shar is attached and if the issues in comment 2 are address it should be good to go. If you need an additional pair of eyes just ping me
Comment 9 jjuanino 2016-03-05 21:57:18 UTC
Hi,

I submit a new shar file, addressing the points suggested by koobs.

py-libacl is not required by borg at all; it has been removed from Makefile

I have added a check to ensure that openssl version is greather or equal to 1.0.0.

I have added a knob to support fuse mount backup borgs. This requires devel/py-llfuse (Bug 203759), thus this port cannot be committed before that port.


Regards
Comment 10 jjuanino 2016-03-05 21:58:19 UTC
Created attachment 167747 [details]
New makefile version
Comment 11 jjuanino 2016-03-05 22:52:26 UTC
Created attachment 167748 [details]
1.0.0 has been release a few hours ago

Hi,

1.0.0 version has been released a few hours ago. I have updated the shar file. Regards
Comment 12 Raphael Kubo da Costa freebsd_committer 2016-03-06 16:39:53 UTC
Given what was mentioned in comment #9, I'm marking this as depending on bug 203759.
Comment 13 Raphael Kubo da Costa freebsd_committer 2016-03-28 18:46:54 UTC
The 1.0.0 shar failed to build here:

===>   py34-borgbackup-1.0.0 depends on package: py34-setuptools_scm>=1.7 - not found
*** Error code 1

That's because devel/py-setuptools_scm will build with Python 2.7 by default. koobs, what's the recommended approach here?

As for the OpenSSL bit in the Makefile: don't you actually need OpenSSL at runtime too? borg/crypto.pyx just seems to be a wrapper around some OpenSSL calls. In this case, you need to add USE_OPENSSL=yes and then set WITH_OPENSSL_PORT=yes when on FreeBSD 9.
Comment 14 jjuanino 2016-03-29 20:39:40 UTC
About openssl:

USE_OPENSSL=yes is already set in the Makefile, but you have to ensure that openssl version is greater or equal to 1.0.0. I do not know the best way to indicate such requirement in the Makefile. I wrote:



USE_OPENSSL=    yes
...............
# borg requires openssl>=1.0.0
.if ${OSVERSION} < 1000015
BUILD_DEPENDS+= openssl>=1.0:${PORTSDIR}/security/openssl
.endif


Regards
Comment 15 Raphael Kubo da Costa freebsd_committer 2016-03-30 12:21:57 UTC
(In reply to jjuanino from comment #14)
> I do not know the best way to
> indicate such requirement in the Makefile. I wrote:
> 
> USE_OPENSSL=    yes
> ...............
> # borg requires openssl>=1.0.0
> .if ${OSVERSION} < 1000015
> BUILD_DEPENDS+= openssl>=1.0:${PORTSDIR}/security/openssl
> .endif
See my suggestion in comment #13:

> borg/crypto.pyx just seems to be a wrapper around some OpenSSL calls. In this case, you need to add USE_OPENSSL=yes and then set WITH_OPENSSL_PORT=yes when on FreeBSD 9.

I literally meant it: you should replace your BUILD_DEPENDS line with WITH_OPENSSL_PORT=yes. It takes care of making security/openssl a build and runtime dependency which is very likely what you want.
Comment 16 jjuanino 2016-03-31 07:50:44 UTC
Yes, I have understood your suggestion, but I only wanted to ensure that openssl>=1.0.0 in ports when ${OSVERSION} < 1000015. 

I am seeing that port version is greather or equal to 1.0.0 since Apr 2 18:42:11 2010... thus it is fine that are saying about (sorry for the noise)

Please go ahead with your proposal:

.if ${OSVERSION} < 1000015
WITH_OPENSSL_PORT=yes
.endif

Regards
Comment 17 jjuanino 2016-04-03 11:21:47 UTC
Hi,

devel/py-llfuse port version is 1.0.

According to setup.py, borg requires llfuse < 1.0, but I have tested borg with 1.0 llfuse version and works well. I have raised the question to upstream 

https://github.com/borgbackup/borg/issues/828

but I am almost sure you can replace 

FUSE_RUN_DEPENDS=       ${PYTHON_PKGNAMEPREFIX}llfuse<1.0:devel/py-llfuse

with

FUSE_RUN_DEPENDS=       ${PYTHONPREFIX_SITELIBDIR}/llfuse.so:${PORTSDIR}/devel/py-llfuse


Regards
Comment 18 Raphael Kubo da Costa freebsd_committer 2016-04-03 11:25:39 UTC
Thanks for taking this upstream. From my point of view, the only thing we still need to solve is the fact that there's no Python 3 version of devel/py-setuptools_scm in the ports tree, which leads to what I said in comment #13.

I guess we will need to follow koobs' advice from comment #2 and create per-Python versions of that port, as is the case for devel/py-setuptools.
Comment 19 jjuanino 2016-04-03 12:06:50 UTC
Uptream has confirmed to me that fuse version <1.0 requirement can be dropped.
Comment 20 Raphael Kubo da Costa freebsd_committer 2016-04-03 12:37:31 UTC
I've sent an email to the freebsd-python mailing list to confirm whether we really need to create 12 new ports because of the Python version issues I've outlined: https://lists.freebsd.org/pipermail/freebsd-python/2016-April/010144.html
Comment 21 jjuanino 2016-04-14 20:38:13 UTC
Hi, I like the suggestion of

https://lists.freebsd.org/pipermail/freebsd-python/2016-April/010151.html

Could you commit with de NO_PACKAGE option?
Comment 22 jjuanino 2016-04-16 09:02:53 UTC
Created attachment 169366 [details]
1.0.1 has been release. I attach the new shar file

This new shar incorporates the NO_PACKAGE flag.
Comment 23 commit-hook freebsd_committer 2016-04-16 17:43:56 UTC
A commit references this bug:

Author: rakuco
Date: Sat Apr 16 17:43:11 UTC 2016
New revision: 413469
URL: https://svnweb.freebsd.org/changeset/ports/413469

Log:
  New port: archivers/py-borgbackup.

  BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it
  supports compression and authenticated encryption.

  The main goal of Borg is to provide an efficient and secure way to backup data.
  The data deduplication technique used makes Borg suitable for daily backups
  since only changes are stored. The authenticated encryption technique makes it
  suitable for backups to not fully trusted targets.

  WWW: https://pypi.python.org/pypi/borgbackup

  Note: NO_PACKAGE=yes had to be set because this port only works with Python 3
  and depends on other Python ports which are currently only built with Python
  2.7 by default. Discussed in the freebsd-python mailing list:
  https://lists.freebsd.org/pipermail/freebsd-python/2016-April/010144.html

  PR:		207715
  Submitted by:	Jos? Garc?a Juanino <jjuanino@gmail.com>

Changes:
  head/archivers/Makefile
  head/archivers/py-borgbackup/
  head/archivers/py-borgbackup/Makefile
  head/archivers/py-borgbackup/distinfo
  head/archivers/py-borgbackup/pkg-descr
Comment 24 Raphael Kubo da Costa freebsd_committer 2016-04-16 17:44:39 UTC
Finally committed, thanks for the patience!
Comment 25 Raphael Kubo da Costa freebsd_committer 2016-04-16 17:59:16 UTC
NO_PACKAGE was replaced with a different check in https://svnweb.freebsd.org/changeset/ports/413471
Comment 26 Kurt Jaeger freebsd_committer 2016-04-22 18:05:18 UTC
Bryan has an issue for poudriere that describes the problem with the
build flags:

https://github.com/freebsd/poudriere/issues/259