Created attachment 176889 [details] Patch adding databases/pg_partman This is a new port, built with PostgreSQL 9.4, 9.5, and 9.6, with each PostgreSQL version and the pg_partman port being built with 9.3, 10.1, 10.2, 10.3, and 11.0 RELEASE amd64. Partitioning functionality was tested partition a single PostgreSQL table of 300GB with 1.2 billion rows on 9.6. All seems to have worked well. Please let me know if further changes or updates are needed. Thank you. =)
Some problem: 1 WANT_PGSQL_VER= 94 should be WANT_PGSQL_VER= 9.4+ ? 2 Depend databases/py-psycopg2 default use postgresql-9.3, it would conflicts with databases/pg_partman. wen
Created attachment 176898 [details] Updated patch to add databases/pg_partman
(In reply to Wen Heping from comment #1) Hello Wen. Thanks for looking. After your comment, I double-checked this was correct, and looked over a few other ports, and noticed that starting line 632 of the CHANGES file that I should have done something like this instead: WANT_PGSQL_VER=91+ becomes USES=pgsql:9.1+ I have updated the patch with that change. Secondly, database/py-psycopg2 seems to only have a default of USES=pgsql and I thought that defaulted to the DEFAULT_VERSION set in your /etc/make.conf file. Since I specified multiple versions when I tested this with poudriere testport, installs with 9.3 and earlier conflicted on the postgresql-client package (9.3 installed vs 9.4 being requested). =======================<phase: run-depends >============================ ===> pg_partman-2.6.1 depends on file: /usr/local/lib/python2.7/site-packages/psycopg2/__init__.py - not found ===> Installing existing package /packages/All/py27-psycopg2-2.6.1_1.txz [11point0-RELEASE-amd64-development] Installing py27-psycopg2-2.6.1_1... [11point0-RELEASE-amd64-development] `-- Installing postgresql93-client-9.3.15... pkg-static: postgresql93-client-9.3.15 conflicts with postgresql94-client-9.4.10 (installs files into the same place). Problematic file: /usr/local/bin/clusterdb Failed to install the following 1 package(s): /packages/All/py27-psycopg2-2.6.1_1.txz *** Error code 70 Stop. make: stopped in /usr/ports/databases/pg_partman =======================<phase: Interactive >============================ [00:00:28] ====>> Installing packages [00:00:28] ====>> Installing run-depends for databases/pg_partman ===> pg_partman-2.6.1 depends on file: /usr/local/lib/python2.7/site-packages/psycopg2/__init__.py - not found ===> Installing existing package /packages/All/py27-psycopg2-2.6.1_1.txz [11point0-RELEASE-amd64-development] Installing py27-psycopg2-2.6.1_1... [11point0-RELEASE-amd64-development] `-- Installing postgresql93-client-9.3.15... pkg-static: postgresql93-client-9.3.15 conflicts with postgresql94-client-9.4.10 (installs files into the same place). Problematic file: /usr/local/bin/clusterdb Failed to install the following 1 package(s): /packages/All/py27-psycopg2-2.6.1_1.txz *** Error code 70 Stop. make: stopped in /usr/ports/databases/pg_partman Is there a different way the runtime version should be handled? Possibly a CONFLICTS_INSTALL=postgresql9[123]-client ?
the DEFAULT_VERSIONS make.conf variable is what you are looking for. For example, I want PostgreSQL 9.6, so I have in my /etc/make.conf: DEFAULT_VERSIONS=pgsql=9.6 apache=2.4 php=7.0 for all the defaults I set.
(In reply to Larry Rosenman from comment #4) Hello Larry, Thank you for your comment. I did originally set the DEFAULT_VERSIONS in /etc/make.conf for each version in my original runs of Poudriere. I had a make.conf: DEFAULT_VERSIONS+=pgsql=9.6 DEFAULT_VERSIONS+=pgsql=9.5 DEFAULT_VERSIONS+=pgsql=9.4 for each PostgreSQL version and FreeBSD version I tested. I also tested with make.conf files containing: DEFAULT_VERSIONS+=pgsql=9.2 DEFAULT_VERSIONS+=pgsql=9.3 which, combined with USES=pgsql:9.4+, caused a poudriere failure when databases/pg_partman attempted to install postgresql94-client to satisfy the USES requirement, as I had expected. I had no problems building or installing this port with 9.4+, including the additional runtime dependency of databases/py-psycopg2. If there are no additional problems found with this port, I feel it is ready to be committed. I've been running this for 2 months without functionality issues (except for the single documented background worker issue noted in the pkg-message) with 9.6 and 11.0-RELEASE.
Take
For future reference, it's easier to add new ports using shar(1) instead of a patch.
Actually, can you re-package the port as a shar(1) archive for me? Thanks!
Created attachment 178873 [details] pg_partman shar file Sure thing. =) Let me know if you need anything else. =)
Assigning multiple values to DEFAULT_VERSIONS for the same port doesn't make sense, there framework only supports one default version (there is no such thing as a "backup default version"). databases/py-psycopg2 just takes the default PostgreSQL version, which is currently 9.3 as defined in Mk/bsd.default-versions.mk. Indeed, I get (with an unmodified DEFAULT_VERSIONS): rene@e17:/usr/ports/databases/pg_partman % make check-sanity ===> pg_partman-2.6.1 cannot install: the port wants postgresql-client version 9.4 9.5 9.6 and you have version 9.3 installed. *** Error code 1 Stop. make: stopped in /usr/home/rene/freebsd/ports/head/databases/pg_partman So if the port really needs PostgreSQL 9.4 or later, we should first update Mk/bsd.default-versions.mk (possibly using an exp-run). Or perhaps it could just use any PostgreSQL version (I don't know the software)?
Some stylistic errors: rene@e17:/usr/ports/databases/pg_partman % portlint -AC WARN: /usr/home/rene/freebsd/ports/head/databases/pg_partman/pkg-plist: [10]: If and only if your port is DOCSDIR-safe (that is, a user can override DOCSDIR when building this port and the port will still work correctly) consider using DOCSDIR macro; if you are unsure if this this port is DOCSDIR-safe, then ignore this warning WARN: /usr/home/rene/freebsd/ports/head/databases/pg_partman/pkg-plist: [11]: If and only if your port is DOCSDIR-safe (that is, a user can override DOCSDIR when building this port and the port will still work correctly) consider using DOCSDIR macro; if you are unsure if this this port is DOCSDIR-safe, then ignore this warning WARN: /usr/home/rene/freebsd/ports/head/databases/pg_partman/pkg-plist: [12]: If and only if your port is DOCSDIR-safe (that is, a user can override DOCSDIR when building this port and the port will still work correctly) consider using DOCSDIR macro; if you are unsure if this this port is DOCSDIR-safe, then ignore this warning WARN: /usr/home/rene/freebsd/ports/head/databases/pg_partman/distinfo: [1]: TIMESTAMP is over 30 days old WARN: /usr/home/rene/freebsd/ports/head/databases/pg_partman/pkg-message: seems to have unnecessary blank lines at the last part. 0 fatal errors and 5 warnings found. (the message about TIMESTAMP is harmless)
so if we have a port that needs PostgreSQL 9.4+, but the default is 9.3, we can't have it in the tree? (this Software REQUIRES 9.4+). I'll clean up style if it's going to go into the tree. What's the gut feeling on updating DEFAULT_VERSIONS to 9.4? (9.3 is getting pretty old). I'm not part of pgsql@ though. adding pgsql@ for input.
(In reply to Rene Ladan from comment #10) Sorry, I guess my testing methodology for this port wasn't clear. I've outlined it below. It says very clearly here: https://github.com/keithf4/pg_partman that this requires 9.4+ as well. I tested each of the following combinations in separate poudriere runs: DEFAULT_VERSIONS+=pgsql=9.2 - 9.3-RELEASE-amd64 9.3-RELEASE-i386 10.1-RELEASE-amd64 10.1-RELEASE-i386 10.2-RELEASE-amd64 10.2-RELEASE-i386 10.3-RELEASE-amd64 10.3-RELEASE-i386 11.0-RELEASE-amd64 11.0-RELEASE-i386 All failed due to USES=pgsql:9.4+ DEFAULT_VERSIONS+=pgsql=9.3 - 9.3-RELEASE-amd64 9.3-RELEASE-i386 10.1-RELEASE-amd64 10.1-RELEASE-i386 10.2-RELEASE-amd64 10.2-RELEASE-i386 10.3-RELEASE-amd64 10.3-RELEASE-i386 11.0-RELEASE-amd64 11.0-RELEASE-i386 All failed due to USES=pgsql:9.4+ DEFAULT_VERSIONS+=pgsql=9.4 - 9.3-RELEASE-amd64 9.3-RELEASE-i386 10.1-RELEASE-amd64 10.1-RELEASE-i386 10.2-RELEASE-amd64 10.2-RELEASE-i386 10.3-RELEASE-amd64 10.3-RELEASE-i386 11.0-RELEASE-amd64 11.0-RELEASE-i386 All versions created a working poudriere package. DEFAULT_VERSIONS+=pgsql=9.5 - 9.3-RELEASE-amd64 9.3-RELEASE-i386 10.1-RELEASE-amd64 10.1-RELEASE-i386 10.2-RELEASE-amd64 10.2-RELEASE-i386 10.3-RELEASE-amd64 10.3-RELEASE-i386 11.0-RELEASE-amd64 11.0-RELEASE-i386 All versions created a working poudriere package. DEFAULT_VERSIONS+=pgsql=9.6 - 9.3-RELEASE-amd64 9.3-RELEASE-i386 10.1-RELEASE-amd64 10.1-RELEASE-i386 10.2-RELEASE-amd64 10.2-RELEASE-i386 10.3-RELEASE-amd64 10.3-RELEASE-i386 11.0-RELEASE-amd64 11.0-RELEASE-i386 All versions created a working poudriere package. I never set multiple default versions in the same make.conf for this to work. Each PostgreSQL version was set in a different make.conf that I symlinked for different poudriere runs. The package was installed and tested in iocage jails using the same PostgreSQL version and FreeBSD version after they were built, to make sure that the package installed and the extension could be created. 9.6 was tested more heavily because that was what I was currently running, and I needed to partition a large table. Sorry for any confusion my descriptions may have caused.
Created attachment 179364 [details] pg_partman shar updated to 2.6.3 Updated to 2.6.3 which according to: https://github.com/keithf4/pg_partman/commit/6d5d0475eabea9b80d66930f47b4dfef3d94dda9 has fixed the background worker bug with PostgreSQL 9.6. I have also fixed all the portlint -AC warnings. Built per the previous testing methodology against 10.3-RELEASE and 11.0-RELEASE (i386 and amd64) and PostgreSQL 9.4, 9.5, and 9.6.
Created attachment 181016 [details] pg_partman shar updated to 2.6.4 Updated to 2.6.4. Tested per previous methods.
Now that 9.5 is the default PostgreSQL version in the 2017Q3 branch, can this please be revisited? =)
Created attachment 184810 [details] pg_partman shar updated to 3.0.2
testbuild@work
Committed, thanks!
A commit references this bug: Author: pi Date: Fri Aug 11 09:26:26 UTC 2017 New revision: 447749 URL: https://svnweb.freebsd.org/changeset/ports/447749 Log: New port: databases/pg_partman pg_partman is an extension to create and manage both time-based and serial-based table partition sets. Sub-partitoning is also supported. Child table & trigger function creation is all managed by the extension itself. Tables with existing data can also have their data partitioned in easily managed smaller batches. Optional retention policy can automatically drop partitions no longer needed. A background worker (BGW) process is included to automatically run partition maintenance without the need of an external scheduler (cron, etc) in most cases. WWW: https://github.com/keithf4/pg_partman PR: 214417 Submitted by: Lacey Powers <lacey.leanne@gmail.com> Reviewed by: ler Changes: head/databases/Makefile head/databases/pg_partman/ head/databases/pg_partman/Makefile head/databases/pg_partman/distinfo head/databases/pg_partman/pkg-descr head/databases/pg_partman/pkg-plist