Bug 203760 - [NEW PORT] net/s3ql: File system that stores all its data online using cloud storage services
Summary: [NEW PORT] net/s3ql: File system that stores all its data online using cloud ...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Kurt Jaeger
URL: https://github.com/niklaas/freebsd-ports
Keywords: needs-qa, patch
Depends on: 203733 203759
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-14 06:24 UTC by Niklaas Baudet von Gersdorff
Modified: 2017-04-14 18:59 UTC (History)
6 users (show)

See Also:
me: maintainer-feedback+


Attachments
s3ql.shar (3.45 KB, application/x-shar)
2015-10-14 06:24 UTC, Niklaas Baudet von Gersdorff
no flags Details
Shar file (3.53 KB, application/x-shar)
2015-10-19 11:51 UTC, Niklaas Baudet von Gersdorff
me: maintainer-approval+
Details
Output of `poudriere testport` (40.17 KB, text/plain)
2015-10-19 11:53 UTC, Niklaas Baudet von Gersdorff
no flags Details
Shar file for version 2.18 (2.97 KB, text/plain)
2016-06-13 14:18 UTC, Niklaas Baudet von Gersdorff
no flags Details
testport log for version 2.18 (39.24 KB, text/plain)
2016-06-13 14:19 UTC, Niklaas Baudet von Gersdorff
no flags Details
s3ql.shar version 2.21 (2.97 KB, application/x-shar)
2017-04-01 18:51 UTC, Niklaas Baudet von Gersdorff
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Niklaas Baudet von Gersdorff 2015-10-14 06:24:44 UTC
Created attachment 162014 [details]
s3ql.shar

I ported this together with devel/py-dugong and devel/llfuse.

---

S3QL is a file system that stores all its data online using storage
services like Google Storage, Amazon S3, or OpenStack. S3QL effectively
provides a hard disk of dynamic, infinite capacity that can be accessed
from any computer with internet access running Linux, FreeBSD or OS-X.

S3QL is a standard conforming, full featured UNIX file system that is
conceptually indistinguishable from any local file system. Furthermore,
S3QL has additional features like compression, encryption, data
de-duplication, immutable trees and snapshotting which make it
especially suitable for online backup and archival.

S3QL is designed to favor simplicity and elegance over performance and
feature-creep. Care has been taken to make the source code as readable and
serviceable as possible. Solid error detection and error handling have been
included from the very first line, and S3QL comes with extensive automated test
cases for all its components.

WWW: https://bitbucket.org/nikratio/s3ql/
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2015-10-14 06:35:23 UTC
Don't forget to set the "Depends on" and "Blocks" fields to point to the correct dependent issues :)

Also, COMMENT is too long (potlint should pick this issue up)

As per my comment in Bug  203759, please run this port through QA tool (portlint/poudriere) and attach results
Comment 2 Niklaas Baudet von Gersdorff 2015-10-15 14:15:28 UTC
(In reply to Kubilay Kocak from comment #1)
I am amazed by "Depends on" and "Blocks". Really cool feature. :)

I've finished updating the dependencies and will update this submission most probably next week.
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2015-10-15 14:23:30 UTC
In the meantime (while you update this submission) - please obsolete the existing attachment 162014 [details] so that it doesnt get accidentally committed as is :)
Comment 4 Niklaas Baudet von Gersdorff 2015-10-15 14:35:21 UTC
(In reply to Kubilay Kocak from comment #3)
Of course, makes sense. :) Thanks for your patience and introduction concerning the workflow here.
Comment 5 Niklaas Baudet von Gersdorff 2015-10-19 11:51:09 UTC
Created attachment 162204 [details]
Shar file
Comment 6 Niklaas Baudet von Gersdorff 2015-10-19 11:53:09 UTC
Created attachment 162205 [details]
Output of `poudriere testport`

Looks fine; despite 9 compiler warnings that I will submit upstream.

---

cc -DNDEBUG -O2 -pipe -fstack-protector -fno-strict-aliasing -I/usr/local/include -fPIC -I/usr/local/include/python3.4m -c src/s3ql/deltadump.c -o build/temp.freebsd-10.1-RELEASE-p22-amd64-3.4/src/s3ql/deltadump.o -Wall -Wextra -Wconversion -Wsign-compare -Werror=conversion -Werror=sign-compare
src/s3ql/deltadump.c:11414:32: warning: unused function '__Pyx_PyUnicode_FromString' [-Wunused-function]
static CYTHON_INLINE PyObject* __Pyx_PyUnicode_FromString(const char* c_str) {
                               ^
src/s3ql/deltadump.c:9065:32: warning: unused function '__Pyx_GetItemInt_List_Fast' [-Wunused-function]
static CYTHON_INLINE PyObject *__Pyx_GetItemInt_List_Fast(PyObject *o, Py_ssize_t i,
                               ^
src/s3ql/deltadump.c:9080:32: warning: unused function '__Pyx_GetItemInt_Tuple_Fast' [-Wunused-function]
static CYTHON_INLINE PyObject *__Pyx_GetItemInt_Tuple_Fast(PyObject *o, Py_ssize_t i,
                               ^
src/s3ql/deltadump.c:9293:26: warning: unused function '__Pyx_PyBytes_Equals' [-Wunused-function]
static CYTHON_INLINE int __Pyx_PyBytes_Equals(PyObject* s1, PyObject* s2, int equals) {
                         ^
src/s3ql/deltadump.c:9992:28: warning: unused function '__Pyx_CyFunction_InitDefaults' [-Wunused-function]
static CYTHON_INLINE void *__Pyx_CyFunction_InitDefaults(PyObject *func, size_t size, int pyobjects) {
                           ^
src/s3ql/deltadump.c:10001:27: warning: unused function '__Pyx_CyFunction_SetDefaultsTuple' [-Wunused-function]
static CYTHON_INLINE void __Pyx_CyFunction_SetDefaultsTuple(PyObject *func, PyObject *tuple) {
                          ^
src/s3ql/deltadump.c:10006:27: warning: unused function '__Pyx_CyFunction_SetDefaultsKwDict' [-Wunused-function]
static CYTHON_INLINE void __Pyx_CyFunction_SetDefaultsKwDict(PyObject *func, PyObject *dict) {
                          ^
src/s3ql/deltadump.c:10011:27: warning: unused function '__Pyx_CyFunction_SetAnnotationsDict' [-Wunused-function]
static CYTHON_INLINE void __Pyx_CyFunction_SetAnnotationsDict(PyObject *func, PyObject *dict) {
                          ^
src/s3ql/deltadump.c:11104:27: warning: function '__Pyx_PyInt_As_long' is not needed and will not be emitted [-Wunneeded-internal-declaration]
static CYTHON_INLINE long __Pyx_PyInt_As_long(PyObject *x) {
                          ^
9 warnings generated.
Comment 7 Martin Wilke freebsd_committer 2015-12-24 18:54:14 UTC
Hi,

Any update on this?
Comment 8 Niklaas Baudet von Gersdorff 2015-12-25 21:24:03 UTC
Hi Martin,

concerning the warnings? Please see the submission of the warnings upstream here: https://bitbucket.org/nikratio/s3ql/issues/168/cc-compiler-warnings-on-freebsd I am in contact with the maintainer. He is in need of a FreeBSD built environment to work on the code. I actually want to offer him such an environment in form of a jail on one of my machines. Because of other projects, I haven't had time to do so yet but I am still keen to put this forward.

As far as I can remember, the port worked fine despite the warnings. I will double-check that and work on the warnings (together with the maintainer) at the end of January. Unfortunately, I won't find time for that before.

Best,

Niklaas
Comment 9 Martin Wilke freebsd_committer 2015-12-26 07:24:44 UTC
Hi Niklaas,

The warnings are ok to me, but the fail of FreeBSD 9.X is not ok, either we mark this port broken for 9X and bring it in, or we wait till there is a fix for 9.X.

- Martin
Comment 10 Niklaas Baudet von Gersdorff 2015-12-26 07:52:56 UTC
(In reply to Martin Wilke from comment #9)

Hi Martin,

ah, I'm sorry, I didn't know about the fail of 9.X. It's one of the first ports that I committed. Probably I've overlooked something here. Where can I have a look at the errors?

I don't mind whether to bring it in or not. Anyway I would take a closer look on it again in January/February and see that it's working on 9.X too. Unfortunately, I am too busy at the moment.

Maybe it makes sense to bring it in now to hope for some further feedback? I could address this in January/February too. But if it's common to wait until a port works on every system before bringing it in, we should stick to that.

Best, Niklaas
Comment 11 Niklaas Baudet von Gersdorff 2016-01-24 20:53:24 UTC
Hi Martin,

probably my comment got lost over Christmas days. Have you had a closer look on this? What's exactly causing the port to fail on 9.x?

Best,

Niklaas
Comment 12 Martin Wilke freebsd_committer 2016-03-14 02:30:58 UTC
back to pool
Comment 13 Kurt Jaeger freebsd_committer 2016-03-22 05:30:22 UTC
testbuilds@work
Comment 14 Kurt Jaeger freebsd_committer 2016-03-22 17:36:57 UTC
Testbuilds all somehow depend on py2.7 and fail, see

http://people.freebsd.org/~pi/logs/net__s3ql-10a

Yes, USES=python:3.3+ is set in the Makefile. Any ideas what's going wrong ?
Comment 15 Niklaas Baudet von Gersdorff 2016-03-22 19:52:06 UTC
Kurt, thanks for taking up on this. The last time I tested this was months ago, so I might be wrong about the following. I'll test it as soon as I can but maybe this will point towards the correct direction.

As far as I can remember, poudriere testport cannot cope with the different python version/dependency on its own. I think someone explained this to me on #bsdports, but I am not sure. That's why I had to put

DEFAULT_VERSIONS=python=3.4 python3=3.4

in make.conf that poudriere uses to build net/s3ql.

When I was told that I was wondering how then it will be possible to build the package later without manual intervention in make.conf. But I assumed that it's just *testport* that cannot cope with the version/dependency. In addition, as far as I can remember, it built fine with bulk (otherwise it wouldn't be here, I guess. :)

I'll try to check the port again as soon as I can and keep you updated. Maybe this is already some help? It will be great if we get it committed eventually.
Comment 16 Kurt Jaeger freebsd_committer 2016-03-22 20:24:17 UTC
Btw, I tested 2.17.1 (the most recent version).

Yes, using DEFAULT_VERSIONS=python=3.4 python3=3.4 in make.conf works,
my question is: Is there a way to enforce it with the make.conf settings.

From 

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207715#c2

I take it that this 3.x enforcing can't be done without seperate py34-xxx ports.
Comment 17 Niklaas Baudet von Gersdorff 2016-03-22 20:32:59 UTC
(In reply to Kurt Jaeger from comment #16)

So there is no way/reason to commit this until there are py34-* ports?
Comment 18 Kurt Jaeger freebsd_committer 2016-03-22 20:35:23 UTC
I ping'ed koobs and wait for his feedback.
Comment 19 Shane 2016-03-23 07:07:56 UTC
To build a port and it's dependencies using python 3.5 when 2.7 is the default you can adjust the defaults in make.conf or you can use 

setenv PYTHON_VERSION 3.5; make

This currently fails in poudriere as each port is built by itself, so they don't inherit the PYTHON_VERSION from the port triggering the dependency. There is work being done to remedy this - https://www.freebsd.org/news/status/report-2015-10-2015-12.html#Supporting-Variants-in-the-Ports-Framework

Personally I use multiple poudriere sets (-x setname) to build python modules, each set name gets a py3x suffix and has it's own mypkgpy3x-make.conf with appropriate defaults set. My config for pkg contains the multiple pkg repo locations and when I pkg install devel/py-tox it finds and installs the packages for each python version.

Note that to get python 3.x packages that will install on a system that has 2.7 as default (which is having scripts in bin installed correctly) you need to set -

DEFAULT_VERSIONS= python=2.7 python2=2.7 python3=3.5
.if ${.CURDIR:M*/py*}
PYTHON_VERSION=3.5
.endif

Not every port builds if PYTHON_VERSION is set globally (for lack of py3 support). I have been getting better results with the CURDIR test.

Obviously not a solution for the public pkg repos. Although I don't know if we would have any luck getting extra repos setup for multiple python versions. Or syncing py3x packages from another build into the public repo...

Until the variant work is completed you may have to consider making users manually install from ports. That could be achieved by adding "NO_PACKAGE=Requires python 3.x to be set as default" - if you have that within an .if ${PYTHON_MAJOR_VER} == 3 then it will still package if the defaults have been set.
Comment 20 Niklaas Baudet von Gersdorff 2016-03-23 15:33:24 UTC
(In reply to FreeBSD from comment #19)
Thanks for the detailed outline. I'll try to implement and testport the suggestions within the next days and update the shar file correspondingly.

This way, at least, the port can make it into the port system.
Comment 21 Niklaas Baudet von Gersdorff 2016-06-13 14:18:33 UTC
Created attachment 171384 [details]
Shar file for version 2.18

Please excuse, I was occupied with other projects.

I attached a new version of the shar file that includes the recommendation concerning NO_PKG (thank you!) and also ports a newer version (2.18 instead of 2.15).
Comment 22 Niklaas Baudet von Gersdorff 2016-06-13 14:19:04 UTC
Created attachment 171385 [details]
testport log for version 2.18
Comment 23 Kurt Jaeger freebsd_committer 2016-06-14 16:55:54 UTC
testbuilds@work
Comment 24 Niklaas Baudet von Gersdorff 2017-04-01 18:51:07 UTC
Created attachment 181390 [details]
s3ql.shar version 2.21

This is in progress for some while now. I haven't looked into this for a long time too, but really would like to get the port added to the ports tree.

The new shar file is for the most recent version of s3ql: 2.21.

Portlint is fine despite some complaints:

WARN: Makefile: no ftp/http mirror in MASTER_SITES for users behind a proxy.
WARN: Makefile: wrong dependency value for BUILD_DEPENDS. BUILD_DEPENDS requires 2 or 3 colon-separated tuples.
WARN: Makefile: wrong dependency value for RUN_DEPENDS. RUN_DEPENDS requires 2 or 3 colon-separated tuples.
WARN: Makefile: extra item placed in the *_DEPENDS section, for example, "MY_DEPENDS".
0 fatal errors and 4 warnings found.

This should be okay, I guess.

I have a bug report running upstream (https://bitbucket.org/nikratio/s3ql/issues/254/chmod-corrupts-file-system). Nevertheless, pushing this in the ports tree maybe eases checking whether this is a general issue on FreeBSD.
Comment 25 commit-hook freebsd_committer 2017-04-14 18:59:15 UTC
A commit references this bug:

Author: swills
Date: Fri Apr 14 18:58:45 UTC 2017
New revision: 438541
URL: https://svnweb.freebsd.org/changeset/ports/438541

Log:
  net/s3ql: create port

  S3QL is a file system that stores all its data online using storage
  services like Google Storage, Amazon S3, or OpenStack. S3QL effectively
  provides a hard disk of dynamic, infinite capacity that can be accessed
  from any computer with internet access running Linux, FreeBSD or OS-X.

  WWW: https://bitbucket.org/nikratio/s3ql

  PR:		203760
  Submitted by:	Niklaas Baudet von Gersdorff <stdin@niklaas.eu> (with changes)

Changes:
  head/net/Makefile
  head/net/s3ql/
  head/net/s3ql/Makefile
  head/net/s3ql/distinfo
  head/net/s3ql/pkg-descr
Comment 26 Steve Wills freebsd_committer 2017-04-14 18:59:47 UTC
Committed, thanks!