Created attachment 210487 [details] patch to fix the bug See the eamil thread that starts here: https://lists.freebsd.org/pipermail/freebsd-current/2019-December/075018.html Basically, compiling with -DWITH_OPENSSL and -DWITHOUT_OPENSSL generates a binary for factor(6), which gives inconsistent results for hexadecimal input. The patch fixes the problem. Here's a svn log entry. * usr.bin/factor/factor.6: . Update documentation to note that hexadecimal strings are accepted. . Document that a hexadecimal number can have an optional 0x or 0X prefix. . Document that a 0 value in interactive mode terminates factor(6). . Fix the maximum value for 'stop' in primes(6). . While here, spell "white-space" as "whitespace" and "non-digit" as "nondigit". * usr.bin/factor/factor.c: . Include stdbool to get acces to bool type. . Use consistent style for function prototypes. . New function. is_hex_str() looks for the longest substring and determines if it is a hexadecimal number. . New function. Factor (pun intended) out common code into convert_str2bn(). . For the WIHTOUT_OPENSSL case, make BN_dec2bn() and BN_hex2bn() return 0 on error like their OpenSSL counterparts. * usr.bin/primes/primes.c: . Fix comment.
Thanks for the submission. Garance Alistair Drosehn or I will get it reviewed and committed.
A commit references this bug: Author: gad Date: Sun Jan 12 20:25:12 UTC 2020 New revision: 356666 URL: https://svnweb.freebsd.org/changeset/base/356666 Log: Fix the way 'factor' behaves when using OpenSSL to match the description of how it works when not compiled with OpenSSL. Also, allow users to specify a hexadecimal number by using a prefix of '0x'. Before this, users could only specify a hexadecimal value if that value included a hex digit ('a'-'f') in the value. PR: 243136 Submitted by: Steve Kargl Reviewed by: gad MFC after: 3 weeks Changes: head/usr.bin/factor/factor.6 head/usr.bin/factor/factor.c head/usr.bin/primes/primes.c
Suggested change made. Will close after MFC.
MFC'ed to 12-stable with -r360835.