Bug 148150 - Poor file(1) performance
Summary: Poor file(1) performance
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 8.1-PRERELEASE
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-06-26 00:00 UTC by Peter Jeremy
Modified: 2018-01-08 04:14 UTC (History)
0 users

See Also:


Attachments
file.diff (814 bytes, patch)
2010-06-26 00:00 UTC, Peter Jeremy
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Jeremy 2010-06-26 00:00:09 UTC
	I recently had reason to run file(1) on a large number of
	files and felt the performance wasn't up to par.  When I
	investigated further, I found that about 95% of the runtime
	related to the two regex's to recognize REXX files:

# OS/2 batch files are REXX. the second regex is a bit generic, oh well
# the matched commands seem to be common in REXX and uncommon elsewhere
100	regex/c =3D^[\ \t]{0,10}call[\ \t]{1,10}rxfunc OS/2 REXX batch file text
100	regex/c =3D^[\ \t]{0,10}say\ ['"]	     OS/2 REXX batch file text

	Since REXX files are not present in my environment, I can
	avoid the issue by just commenting out the offending lines.
	Someone with more expertise in magic(5) might be able to
	suggest a better fix.

	I have tried reporting this to the upstream maintainers and
`	received a "not interested" response.

Fix: The following just comments out the REXX test.
How-To-Repeat: 	Copy /usr/share/misc/magic to magic.old
	Apply the equivalent of the below patch to create magic.new
	time(1) file(1) on the same set of files using magic.old and magic.new

	Using my home directory on my i386 netbook, I get:
file -m magic.new * > /dev/null  1.42s user 0.13s system 98% cpu 1.576 total
file -m magic.new * > /dev/null  1.35s user 0.10s system 98% cpu 1.469 total
file -m magic.new * > /dev/null  1.35s user 0.10s system 98% cpu 1.470 total
file -m magic.old * > /dev/null  33.35s user 0.11s system 98% cpu 34.055 total
file -m magic.old * > /dev/null  33.12s user 0.14s system 98% cpu 33.714 total
file -m magic.old * > /dev/null  33.08s user 0.11s system 98% cpu 33.606 total

	Using my home directory on my amd64 desktop, I get:
file -m magic.new * > /dev/null  2.18s user 0.41s system 28% cpu 9.111 total
file -m magic.new * > /dev/null  2.11s user 0.49s system 24% cpu 10.707 total
file -m magic.new * > /dev/null  2.05s user 0.56s system 23% cpu 10.989 total
file -m magic.old * > /dev/null  28.54s user 0.51s system 78% cpu 37.088 total
file -m magic.old * > /dev/null  28.54s user 0.52s system 89% cpu 32.575 total
file -m magic.old * > /dev/null  28.71s user 0.47s system 99% cpu 29.371 total

	The poorer wallclock performance on my amd64 is because it's
	running ZFS without adequate RAM whereas my netbook is UFS on SSD
	and the actual directory contents are completely different.
Comment 1 Garrett Cooper 2010-06-26 01:54:37 UTC
FWIW I think that this is more indicative of poor regexp(3)
performance or possibly tighter constraints placed on the regexp
compiler / parser to do the act of parsing the string.

Not saying that what you proposed isn't valid, but it's definitely an
interesting note that should be brought up to the upstream folks.

Thanks!
-Garrett
Comment 2 Peter Jeremy 2010-06-26 03:43:58 UTC
On 2010-Jun-25 17:54:37 -0700, Garrett Cooper <yanefbsd@gmail.com> wrote:
>FWIW I think that this is more indicative of poor regexp(3)
>performance or possibly tighter constraints placed on the regexp
>compiler / parser to do the act of parsing the string.


There are several other regexp's that don't have the same impact.  My
suspicion is that at least some of the problem is the 100 line offset.
My feeling is that moving the existing regexp down a level and
wrapping it with a more efficient test that selects potential REXX
files might be the optimal solution but I don't know either REXX or
magic(5) well enough to suggest one.

-- 
Peter Jeremy
Comment 3 Edwin Groothuis freebsd_committer 2010-10-21 21:53:01 UTC
State Changed
From-To: open->analyzed

The best patch so far without loosing features would be: 

-100    regex/c =^[ t]{0,10}call[ t]{1,10}rxfunc OS/2 REXX batch file text 
-100    regex/c =^[ t]{0,10}say ['"]      OS/2 REXX batch file text 
+100     regex/c =^[ t]+call[ t]+rxfunc OS/2 REXX batch file text 
+100     regex/c =^[ t]+say ['"]           OS/2 REXX batch file text 

which reduces the worst-case time from 1900 ms to 450 ms, full 
removal would reduce it back to 170 ms. Worst case scenario is the 
file aclocal.m4 in the contrib/file directory. 



Comment 4 Edwin Groothuis freebsd_committer 2010-10-21 21:53:01 UTC
Responsible Changed
From-To: freebsd-bugs->obrien

obrien@ wants to pre-approve changes or do them himself. 

obrien@: Let me know if I can commit this one!
Comment 5 Eitan Adler freebsd_committer freebsd_triage 2012-03-25 04:38:14 UTC
Responsible Changed
From-To: obrien->freebsd-bugs

timeout
Comment 6 Eitan Adler freebsd_committer freebsd_triage 2012-03-25 04:42:53 UTC
Responsible Changed
From-To: freebsd-bugs->edwin

timeout
Comment 7 Eitan Adler freebsd_committer freebsd_triage 2018-01-08 04:14:31 UTC
For the following conditions
Product: Base System, Documentation Status: New, Open, In Progress, UNCONFIRMED 
Assignee: Former FreeBSD committer 

Reset to default assignee. Reset status to "Open".