Bug 219165 - [NEW PORT] databases/pgloader3 a data loading tool for PostgreSQL,using the COPY command
Summary: [NEW PORT] databases/pgloader3 a data loading tool for PostgreSQL,using the C...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Torsten Zuehlsdorff
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-05-09 10:04 UTC by Jov
Modified: 2017-08-07 13:47 UTC (History)
3 users (show)

See Also:


Attachments
pgloader port shar file (2.06 KB, text/plain)
2017-05-09 10:04 UTC, Jov
no flags Details
v2 (2.09 KB, text/plain)
2017-05-18 02:21 UTC, Jov
no flags Details
v3,add man doc (2.19 KB, text/plain)
2017-05-22 09:10 UTC, Jov
no flags Details
pgloader3.shar v4 (2.24 KB, text/plain)
2017-05-27 10:34 UTC, Jov
no flags Details
pglader3.shar (2.30 KB, text/plain)
2017-06-09 05:39 UTC, Jov
no flags Details
pgloader3.shar (5.09 KB, text/plain)
2017-07-12 02:21 UTC, Jov
no flags Details
pgloader3.shar (5.25 KB, text/plain)
2017-07-14 09:14 UTC, Jov
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jov 2017-05-09 10:04:11 UTC
Created attachment 182435 [details]
pgloader port shar file

Pgloader loads data into PostgreSQL using the COPY streaming protocol,and
doing so with separate threads for reading and writing data.Asynchronous
IO,compiled to the metal programming language.
pgloader knows how to read data from MySQL,SQLite or dBase databases,and
knows how to discover the schema and properly cast data types over.It also
knows how to read CSV files in several definitions of CSV,and fixed columns
files.

WWW: http://pgloader.io
Comment 1 Jov 2017-05-18 02:21:07 UTC
Created attachment 182680 [details]
v2

v1 depend sbcl which do not support multi-thread for default build.The upstream author advice to use ccl which has multi-thread support and has more efficiently garbage collection,see:
https://github.com/dimitri/pgloader/issues/551
Comment 2 Jov 2017-05-22 09:10:09 UTC
Created attachment 182794 [details]
v3,add man doc

Add man file for pgloader
Comment 3 Jov 2017-05-27 10:34:30 UTC
Created attachment 182960 [details]
pgloader3.shar v4

v4,replace ${MV} with ${INSTALL_PROGRAM}
Comment 4 Jov 2017-06-09 05:39:16 UTC
Created attachment 183350 [details]
pglader3.shar

v5:
1.remove do-build
2.make ccl do not use /root/.cache dir

now portlint clean poudriere testport pass.
Comment 5 Jov 2017-06-15 09:43:29 UTC
New patch move to here for review:
https://reviews.freebsd.org/D11212
Comment 6 Jov 2017-06-20 01:06:51 UTC
ping
Comment 7 Mathieu Arnold freebsd_committer 2017-07-06 13:52:56 UTC
Assignee timeout. Give back to the pool.
Comment 8 Jov 2017-07-06 14:25:54 UTC
hi @tz , this is a very useful tool , please take it at your convenience.
Comment 9 Torsten Zuehlsdorff freebsd_committer 2017-07-11 13:54:00 UTC
Hello Jov,

version 3.4.1 was released last week. Do you mind to update your PR?

Also: there is already databases/pgloader with version 2.3.1. After looking into the GitHub account of the project it seems, that the 2.x branch got its last change five years before today. :D

So i would prefer to see an update of the current port instead of creating a new one. I couldn't find any information about current support of 2.x, but i assume there isn't any. Or do you have other reasons to choose a new port over an update?

Greetings,
Torsten
Comment 10 Jov 2017-07-12 02:21:51 UTC
Created attachment 184291 [details]
pgloader3.shar

update to 3.4.1
Comment 11 Jov 2017-07-12 02:52:52 UTC
(In reply to Torsten Zuehlsdorff from comment #9)
I have updated this PR to 3.4.1.

pgloader3 is not compatible with pgloader2. While pgloader 2.x was written in Python years ago, the author rewrites the pgloader 3.x with common lisp recently. The concept and configuration of pgloader3 are far more different with pgloader2.End user needs to learn the new usage and rewrite and test the configuration before the update. There may be someone still using pgloader2, I do not want to break their work and make them unhappy for this POLA violation. 

Any idea?
Comment 12 Torsten Zuehlsdorff freebsd_committer 2017-07-14 08:44:20 UTC
This seems reasonable enough - even if pgloader2 isn't develop anymore its still usable.

In this case we will need to add PKGNAMESUFFIX for differentiation between the ports.

Also i looked at your patches, especially the Makefile patch. Wouldn't it be cleaner to patch out the "clones" block instead of out-commenting each clone?
Comment 13 Jov 2017-07-14 09:14:16 UTC
Created attachment 184338 [details]
pgloader3.shar

1.use PKGNAMESUFFIX = 3
2.make the patch of Makefile simple. Make no target depends on clones, I think this patch is smaller than comment out the "clones" block.
Comment 14 Torsten Zuehlsdorff freebsd_committer 2017-07-21 11:50:04 UTC
This looks fine on a short glance. I will have time in the beginning of the next week to commit it! :)
Comment 15 Jov 2017-07-21 13:13:23 UTC
(In reply to Torsten Zuehlsdorff from comment #14)
Thanks!
Comment 16 Jov 2017-08-07 10:06:45 UTC
Hi Torsten, please have a look if you have time:)
Comment 17 commit-hook freebsd_committer 2017-08-07 13:47:08 UTC
A commit references this bug:

Author: tz
Date: Mon Aug  7 13:46:51 UTC 2017
New revision: 447494
URL: https://svnweb.freebsd.org/changeset/ports/447494

Log:
  New port: databases/pgloader3

  Pgloader loads data into PostgreSQL using the COPY streaming protocol,and
  doing so with separate threads for reading and writing data.Asynchronous
  IO,compiled to the metal programming language.
  pgloader knows how to read data from MySQL,SQLite or dBase databases,and
  knows how to discover the schema and properly cast data types over.It also
  knows how to read CSV files in several definitions of CSV,and fixed columns
  files.

  WWW: http://pgloader.io

  PR:           219165
  Submitted by: Jov <amutu@amutu.com>

Changes:
  head/databases/Makefile
  head/databases/pgloader3/
  head/databases/pgloader3/Makefile
  head/databases/pgloader3/distinfo
  head/databases/pgloader3/files/
  head/databases/pgloader3/files/patch-Makefile
  head/databases/pgloader3/files/patch-pgloader__wrapper.sh
  head/databases/pgloader3/pkg-descr
Comment 18 Torsten Zuehlsdorff freebsd_committer 2017-08-07 13:47:47 UTC
Finally. I'm sorry it took me so long.