Bug 213614 - optimize copytree_*
Summary: optimize copytree_*
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Mathieu Arnold
URL: https://reviews.freebsd.org/D8283
Keywords:
Depends on:
Blocks: 213746
  Show dependency tree / graph
 
Reported: 2016-10-19 14:38 UTC by Mathieu Arnold
Modified: 2016-11-09 12:16 UTC (History)
3 users (show)

See Also:
mat: exp-run?

Attachments
v1 (1.41 KB, patch)
2016-10-19 14:38 UTC, Mathieu Arnold 		no flags Details | Diff
v2 (1.59 KB, patch)
2016-10-20 15:57 UTC, Mathieu Arnold 		no flags Details | Diff
v3 (1.58 KB, patch)
2016-10-21 14:30 UTC, Mathieu Arnold 		no flags Details | Diff
Show Obsolete (2) View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Arnold freebsd_committer freebsd_triage 2016-10-19 14:38:52 UTC 
Created attachment 175935 [details]
v1
Comment 1 Mathieu Arnold freebsd_committer freebsd_triage 2016-10-19 14:39:45 UTC 
tried it on a few ports, does not seem to break stuff, let's try it for real.
Comment 2 Mathieu Arnold freebsd_committer freebsd_triage 2016-10-20 15:57:36 UTC 
Created attachment 175982 [details]
v2

Switch to using find -E.
Comment 3 Mathieu Arnold freebsd_committer freebsd_triage 2016-10-21 14:30:22 UTC 
Created attachment 176023 [details]
v3

Think about it and remove one find.
Comment 4 Jan Beich freebsd_committer freebsd_triage 2016-11-02 06:39:55 UTC 
While you're hacking on COPYTREE_* can you take a look if bug 199550 can be fixed as well? Ignore if non-trivial.
Comment 5 Mathieu Arnold freebsd_committer freebsd_triage 2016-11-02 07:52:50 UTC 
Ah, that one is non trivial to fix. And I fought with it quite a bit before sending the update here.
I think it best to document how to pass wildcards so that they work in the end.
Comment 6 Antoine Brodin freebsd_committer freebsd_triage 2016-11-09 09:01:02 UTC 
Exp-run looks fine.
Comment 7 commit-hook freebsd_committer freebsd_triage 2016-11-09 12:15:06 UTC 
A commit references this bug:

Author: mat
Date: Wed Nov  9 12:14:47 UTC 2016
New revision: 425793
URL: https://svnweb.freebsd.org/changeset/ports/425793

Log:
  Optimize COPYTREE_* macros.

  The main problem slowing down those macros was that it was ending up
  calling chmod(1) once for each file and directory.
  It was doing this because it was running find(1) in the source tree and
  needed to chmod in the target tree, so it had to append the full path to
  the chmod(1) calls, which made it impossible to use -exec +.

  Rewrite the -exec calls to call sh and cd into it before running
  chmod(1) and do so on as many files as we can using the -exec + construct.

  While there, optimize a bit more after figuring out that it is possible
  to only use one find(1) to run two -exec for different options.

  Also switch to using the more powerfull modern regular expressions
  instead of the usual basic regular expressions by using find -E.  This
  will allow the use of | which as no basic regular expressions equivalent
  (as in, for example, "*.(html|txt)") as well as the + and ? keywords,
  and back references.

  About the speed gain, I did a few tests, using time's builtin from csh (table
  with all the data available in the review), comparing the home baked versions
  of COPYTREE_SHARE found in ports, it boils down to:

  |  version   |   www/sit (~2k)  | www/mathjax (~30k) |
  +------------+------------------+--------------------+
  | home baked | 0.032u 0.104s    | 0.946u 7.446s      |
  | current    | 0.281u 0.649s    | 5.446u 18.626s     |
  | new        | 0.024u 0.081s    | 0.770u 7.012s      |

  PR:		213614
  Submitted by:	mat
  Exp-run by:	antoine
  Sponsored by:	Absolight
  Differential Revision:	https://reviews.freebsd.org/D8283

Changes:
  head/Mk/bsd.port.mk