Bug 28988

Anders Nordby <anders@fix.no> writes:
> >Description:
> 
> Add tools for generating 160 bit SHA1 and RMD-160 digests.
> 
> Obtained from OpenBSD. I'm sorry to break Ruslan's WARNS= 2, but I haven't
> been able to remove all warnings (yet). Suggestions are very welcome.
> 
> Patches are relative to 5.0-20010618-CURRENT (snapshot from
> current.freebsd.org). Files added: sha1.1 and rmd160.1.
> 
> Yes, I know openssl dgst -whatnot can do this. But the md5 program is heavily
> used, and thereby I think we should have sha1 and rmd160 too.

Used by whom, and why aren't they using openssl(1)?  The latter is
more portable and more up-to-date.  I think md5(1) should be removed
in favor of OpenSSL; certainly we shouldn't be adding new
functionality to it to encourage its use.  The only thing in the way
of replacng md5(1) with oppenssl(1) completely is that openssl(1)'s
output is a little different with regards to spacing.  At the very
least that will break the ports collection, but the latter should be
adjusted to use openssl(1), anyway.

On Sun, Jul 15, 2001 at 05:26:22AM -0700, Dima Dorfman wrote:
>> Yes, I know openssl dgst -whatnot can do this. But the md5 program is heavily
>> used, and thereby I think we should have sha1 and rmd160 too.
> Used by whom, and why aren't they using openssl(1)?

MD5 checksums are quite heavily used in FreeBSD, and AFAIK most people
use the md5 program for generating MD5 fingerprints of files. At least,
I do, and I haven't heard about anyone using openssl(1) for it in
FreeBSD until recently. :-)

> The latter is more portable and more up-to-date.

More portable? md5(1) is just a frontend for libmd. If I make md5(1) use
libcrypto instead, would that be better? And speaking of up-to-date,
libmd must be the real issue here. It's the library that provides the
actual message-digest fingerprinting code, and is based on SSLeay. We
shouldn't be dragging old SSLeay code around when we have OpenSSL in the
base system.

> I think md5(1) should
> be removed in favor of OpenSSL; certainly we shouldn't be adding new 
> functionality to it to encourage its use. The only thing in the way of
> replacng md5(1) with oppenssl(1) completely is that openssl(1)'s
> output is a little different with regards to spacing.

I don't see why we need to remove md5(1). It can be improved, and it's
already an easy to use tool which can support rmd160 and sha1 too.

Regards,

-- 
Anders.

Anders Nordby <anders@fix.no> writes:
> On Sun, Jul 15, 2001 at 05:26:22AM -0700, Dima Dorfman wrote:
> >> Yes, I know openssl dgst -whatnot can do this. But the md5 program is heavily
> >> used, and thereby I think we should have sha1 and rmd160 too.
> > Used by whom, and why aren't they using openssl(1)?
> 
> MD5 checksums are quite heavily used in FreeBSD, and AFAIK most people
> use the md5 program for generating MD5 fingerprints of files. At least,
> I do, and I haven't heard about anyone using openssl(1) for it in
> FreeBSD until recently. :-)
> 
> > The latter is more portable and more up-to-date.
> 
> More portable?

That too.

> md5(1) is just a frontend for libmd. If I make md5(1) use
> libcrypto instead, would that be better?

Why?  What's wrong with:

	ln -s /usr/bin/openssl md5

works for sha, sha1, ripemd160, and a few other algorithms.  The only
thing wrong with it is that openssl doesn't support some of the
options md5(1) does, and has a slightly different output format.

On Mon, Jul 16, 2001 at 10:20:00PM -0700, Dima Dorfman wrote:
>> md5(1) is just a frontend for libmd. If I make md5(1) use
>> libcrypto instead, would that be better?
> Why?  What's wrong with:
> 
> 	ln -s /usr/bin/openssl md5

The fact that md5 dgst -sha1 does not make sence, and that md5(1) is
expected to output differently and have other options.

> works for sha, sha1, ripemd160, and a few other algorithms.  The only
> thing wrong with it is that openssl doesn't support some of the
> options md5(1) does, and has a slightly different output format.

Is there a problem with having both md5/rmd160/sha1(1) and openssl(1)?
As long as we have libmd and they all work/are correct, I think not.

Regards,

-- 
Anders.

Anders Nordby <anders@fix.no> writes:
> On Mon, Jul 16, 2001 at 10:20:00PM -0700, Dima Dorfman wrote:
> >> md5(1) is just a frontend for libmd. If I make md5(1) use
> >> libcrypto instead, would that be better?
> > Why?  What's wrong with:
> > 
> > 	ln -s /usr/bin/openssl md5
> 
> The fact that md5 dgst -sha1 does not make sence, and that md5(1) is
> expected to output differently

The output is a problem; I agree.

> and have other options.

This is not.  Options can be added.  IIRC, the only non-debug option
that's missing is -s.

> > works for sha, sha1, ripemd160, and a few other algorithms.  The only
> > thing wrong with it is that openssl doesn't support some of the
> > options md5(1) does, and has a slightly different output format.
> 
> Is there a problem with having both md5/rmd160/sha1(1) and openssl(1)?
> As long as we have libmd and they all work/are correct, I think not.

It'd be nice to minimize code duplication in the tree.  If openssl(1)
didn't have a different format and had -s, we'd be all set.  The
latter shouldn't be too difficult to add.  The former might be a
problem, but it's solvable.  Perhaps we can convince the OpenSSL folks
to add our format around an #ifdef or something.

On Thu, Jul 19, 2001 at 04:30:30AM -0700, Anders Nordby wrote:
> The following reply was made to PR bin/28988; it has been noted by GNATS.
> 
> From: Anders Nordby <anders@fix.no>
> To: Dima Dorfman <dima@unixfreak.org>
> Cc: FreeBSD-gnats-submit@freebsd.org
> Subject: Re: bin/28988: We need more simple message digesting tools
> Date: Thu, 19 Jul 2001 13:28:57 +0200
> 
>  On Mon, Jul 16, 2001 at 10:20:00PM -0700, Dima Dorfman wrote:
>  >> md5(1) is just a frontend for libmd. If I make md5(1) use
>  >> libcrypto instead, would that be better?
>  > Why?  What's wrong with:
>  > 
>  > 	ln -s /usr/bin/openssl md5
>  
>  The fact that md5 dgst -sha1 does not make sence, and that md5(1) is
>  expected to output differently and have other options.

It doesn't work, either.  If openssl is invoked with an algorithm name,
it accepts only filenames, not commands/options.  Thus, it preserves
md5(1)'s syntax (at least in this respect).


[roam@edge:p1 /usr/src]$ ln -s /usr/bin/openssl /usr/tmp/md5
[roam@edge:p1 /usr/src]$ /usr/tmp/md5 /etc/passwd
MD5(/etc/passwd)= dbcf40cb4677824638143626b514ad4e
[roam@edge:p1 /usr/src]$ /usr/tmp/md5 dgst -sha1 /etc/passwd
dgst: No such file or directory
-sha1: No such file or directory
MD5(/etc/passwd)= dbcf40cb4677824638143626b514ad4e
[roam@edge:p1 /usr/src]$

>  > works for sha, sha1, ripemd160, and a few other algorithms.  The only
>  > thing wrong with it is that openssl doesn't support some of the
>  > options md5(1) does, and has a slightly different output format.
>  
>  Is there a problem with having both md5/rmd160/sha1(1) and openssl(1)?
>  As long as we have libmd and they all work/are correct, I think not.

I think that there is an idea to remove libmd, since it only duplicates
libcrypto functionality.  Thus, the notion of using openssl(1) as md5(1).

(And before phk jumps at me, yes, I know that libmd was introduced waay
before libcrypto made it into the source tree, and yes, I know that they
have slightly different functionality - actually I even contributed some
of that difference, remember? :P  .. and no, I'm not the one advocating
libmd replacement with libcrypto, just stating that some people are :)

G'luck,
Peter

-- 
Hey, out there - is it *you* reading me, or is it someone else?

Peter Pentchev <roam@orbitel.bg> writes:
> The following reply was made to PR bin/28988; it has been noted by GNATS.
> 
> From: Peter Pentchev <roam@orbitel.bg>
> To: Anders Nordby <anders@fix.no>
> Cc: freebsd-gnats-submit@FreeBSD.org
> Subject: Re: bin/28988: We need more simple message digesting tools
> Date: Thu, 19 Jul 2001 17:05:00 +0300
> 
>  On Thu, Jul 19, 2001 at 04:30:30AM -0700, Anders Nordby wrote:
>  > The following reply was made to PR bin/28988; it has been noted by GNATS.
>  > 
>  > From: Anders Nordby <anders@fix.no>
>  > To: Dima Dorfman <dima@unixfreak.org>
>  > Cc: FreeBSD-gnats-submit@freebsd.org
>  > Subject: Re: bin/28988: We need more simple message digesting tools
>  > Date: Thu, 19 Jul 2001 13:28:57 +0200
>  > 
>  >  On Mon, Jul 16, 2001 at 10:20:00PM -0700, Dima Dorfman wrote:
>  >  >> md5(1) is just a frontend for libmd. If I make md5(1) use
>  >  >> libcrypto instead, would that be better?
>  >  > Why?  What's wrong with:
>  >  > 
>  >  > 	ln -s /usr/bin/openssl md5
>  >  
>  >  The fact that md5 dgst -sha1 does not make sence, and that md5(1) is
>  >  expected to output differently and have other options.
>  
>  It doesn't work, either.  If openssl is invoked with an algorithm name,
>  it accepts only filenames, not commands/options.  Thus, it preserves
>  md5(1)'s syntax (at least in this respect).
>  
>  
>  [roam@edge:p1 /usr/src]$ ln -s /usr/bin/openssl /usr/tmp/md5
>  [roam@edge:p1 /usr/src]$ /usr/tmp/md5 /etc/passwd
>  MD5(/etc/passwd)= dbcf40cb4677824638143626b514ad4e
>  [roam@edge:p1 /usr/src]$ /usr/tmp/md5 dgst -sha1 /etc/passwd
>  dgst: No such file or directory
>  -sha1: No such file or directory
>  MD5(/etc/passwd)= dbcf40cb4677824638143626b514ad4e
>  [roam@edge:p1 /usr/src]$

Actually, it does accept arguments.  Just not the same way openssl(1)
does.  'dgst' is the actual program name inside OpenSSL; you can think
of 'openssl md5' as being a symlink to 'openssl dgst md5'.  When you
make a real symlink called 'md5' to /usr/bin/openssl, you're actually
making a symlink to the 'dgst' program, which is embedded inside
openssl(1).  You can't run other embedded programs, but you can switch
algorithms.  Observe:

	dima@hornet% ln -s /usr/bin/openssl md5
	dima@hornet% ./md5 /HORNET
	MD5(/HORNET)= e3823deaea6ac04928da26c118a36149
	dima@hornet% ./md5 -sha1 /HORNET
	SHA1(/HORNET)= 01539871ab3ece58b874bf47be652508a1e11649

That said, I fail to see why this is a problem.

>  (And before phk jumps at me, yes, I know that libmd was introduced waay
>  before libcrypto made it into the source tree, and yes, I know that they
>  have slightly different functionality - actually I even contributed some
>  of that difference, remember? :P  .. and no, I'm not the one advocating
>  libmd replacement with libcrypto, just stating that some people are :)

I don't know if you think I'm one of these people, but I really have
no opinion on libmd.  As long as things are using it, I have no
problem with it staying.  However, new things probably shouldn't be
implemented using it.  In this case, openssl(1) is almost the perfect
fit: what we need is *already* implemented.  Besides, when the XYZ
hash becomes popular, chances are OpenSSL will implement it quickly;
then, for us to have an xyz(1), all we'd have to do is create another
symlink.  This is certainly better than implementing XYZ in libmd.

At 11:00 AM -0700 7/16/01, Anders Nordby wrote:
>The following reply was made to PR bin/28988; it has been noted by GNATS.
>
>From: Anders Nordby <anders@fix.no>
>To: Dima Dorfman <dima@unixfreak.org>
>Cc: FreeBSD-gnats-submit@freebsd.org
>Subject: Re: bin/28988: We need more simple message digesting tools
>Date: Mon, 16 Jul 2001 19:57:13 +0200
>
>  On Sun, Jul 15, 2001 at 05:26:22AM -0700, Dima Dorfman wrote:
>  >> Yes, I know openssl dgst -whatnot can do this. But the md5 
>program is heavily
>  >> used, and thereby I think we should have sha1 and rmd160 too.
>  > Used by whom, and why aren't they using openssl(1)?
>
>  MD5 checksums are quite heavily used in FreeBSD, and AFAIK most people
>  use the md5 program for generating MD5 fingerprints of files. At least,
>  I do, and I haven't heard about anyone using openssl(1) for it in
>  FreeBSD until recently. :-)
>
>  > The latter is more portable and more up-to-date.
>
>  More portable? md5(1) is just a frontend for libmd. If I make md5(1) use
>  libcrypto instead, would that be better? And speaking of up-to-date,
>  libmd must be the real issue here. It's the library that provides the
>  actual message-digest fingerprinting code, and is based on SSLeay. We
>  shouldn't be dragging old SSLeay code around when we have OpenSSL in the
>  base system.


Over the weekend, I had worked up these patches to md(1) to use 
openssl instead of libmd. I was concerned about maintaining 
compatibility with the existing md in terms of features/options and 
size/location. As has been stated, openssl(1) doesn't have the same 
options and is way too big (dynamically linked!) to put into /sbin.

# ls -l /sbin/md5 /usr/bin/openssl
-r-xr-xr-x  1 root  wheel   56460 Jul  8 11:36 /sbin/md5
-r-xr-xr-x  1 root  wheel  256892 Jul  8 11:38 /usr/bin/openssl
# file /sbin/md5 /usr/bin/openssl
/sbin/md5:        ELF 32-bit LSB executable, Intel 80386, version 1 
(FreeBSD), statically linked, stripped
/usr/bin/openssl: ELF 32-bit LSB executable, Intel 80386, version 1 
(FreeBSD), dynamically linked (uses shared libs), stripped

The patch below is a quick rewrite of md5.c to use libcrypto instead 
of libmd. The size is comparable and libcrypto provides a nice speed 
increase.

Mark

-----------------------
[/sbin/md5 is the old version and ./md5 is the new version]

# ls -l ./md5 /sbin/md5
-rwxr-xr-x  1 root  wheel  57204 Jul 15 13:57 ./md5
-r-xr-xr-x  1 root  wheel  56460 Jul  8 11:36 /sbin/md5
# file ./md5 /sbin/md5
./md5:     ELF 32-bit LSB executable, Intel 80386, version 1 
(FreeBSD), statically linked, stripped
/sbin/md5: ELF 32-bit LSB executable, Intel 80386, version 1 
(FreeBSD), statically linked, stripped

# /sbin/md5 -t ; /sbin/md5 -x ; /sbin/md5 -r md5.c Makefile
MD5 time trial. Digesting 100000 10000-byte blocks ... done
Digest = 766a2bb5d24bddae466c572bcabca3ee
Time = 16 seconds
Speed = 62500000 bytes/second
MD5 test suite:
MD5 ("") = d41d8cd98f00b204e9800998ecf8427e
MD5 ("a") = 0cc175b9c0f1b6a831c399e269772661
MD5 ("abc") = 900150983cd24fb0d6963f7d28e17f72
MD5 ("message digest") = f96b697d7cb7938d525a2f31aaf161d0
MD5 ("abcdefghijklmnopqrstuvwxyz") = c3fcd3d76192e4007dfb496cca67e13b
MD5 
("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789") = 
d174ab98d277d9f5a5611c2c9f419d9f
MD5 
("12345678901234567890123456789012345678901234567890123456789012345678901234567890") 
= 57edf4a22be3c955ac49da2e2107b67a
2a741f9d5f4134df912f3a60a69d471f md5.c
1fc6656efb78f623fdacab2f871c8a82 Makefile

# ./md5 -t ; ./md5 -x ; ./md5 -r md5.c Makefile
MD5 time trial. Digesting 100000 10000-byte blocks ... done
Digest = 766a2bb5d24bddae466c572bcabca3ee
Time = 10 seconds
Speed = 100000000 bytes/second
MD5 test suite:
MD5 ("") = d41d8cd98f00b204e9800998ecf8427e
MD5 ("a") = 0cc175b9c0f1b6a831c399e269772661
MD5 ("abc") = 900150983cd24fb0d6963f7d28e17f72
MD5 ("message digest") = f96b697d7cb7938d525a2f31aaf161d0
MD5 ("abcdefghijklmnopqrstuvwxyz") = c3fcd3d76192e4007dfb496cca67e13b
MD5 
("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789") = 
d174ab98d277d9f5a5611c2c9f419d9f
MD5 
("12345678901234567890123456789012345678901234567890123456789012345678901234567890") 
= 57edf4a22be3c955ac49da2e2107b67a
2a741f9d5f4134df912f3a60a69d471f md5.c
1fc6656efb78f623fdacab2f871c8a82 Makefile



Index: sbin/md5/Makefile
===================================================================
RCS file: /cvs/freebsd/src/sbin/md5/Makefile,v
retrieving revision 1.6
diff -u -r1.6 Makefile
--- sbin/md5/Makefile	2001/06/22 21:38:28	1.6
+++ sbin/md5/Makefile	2001/07/15 21:03:23
@@ -3,7 +3,7 @@

  PROG=	md5

-LDADD+=	-lmd
+LDADD+=	-lcrypto
  DPADD+= ${LIBMD}

  WARNS?=	2
Index: sbin/md5/md5.c
===================================================================
RCS file: /cvs/freebsd/src/sbin/md5/md5.c,v
retrieving revision 1.25
diff -u -r1.25 md5.c
--- sbin/md5/md5.c	2001/06/29 06:21:57	1.25
+++ sbin/md5/md5.c	2001/07/15 21:03:23
@@ -24,13 +24,15 @@

  #include <sys/types.h>
  #include <err.h>
-#include <md5.h>
+#include <openssl/md5.h>
  #include <stdio.h>
  #include <stdlib.h>
  #include <string.h>
  #include <time.h>
  #include <unistd.h>

+#define DIGESTSIZE	16
+#define HEXSIZE		(DIGESTSIZE * 2 + 1)
  /*
   * Length of test block, number of test blocks.
   */
@@ -44,6 +46,8 @@
  static void MDTimeTrial(void);
  static void MDTestSuite(void);
  static void MDFilter(int);
+static char *MDFile(char *, char *);
+static char *Convert2Hex(unsigned char *, char *);
  static void usage(void);

  /* Main driver.
@@ -60,7 +64,7 @@
  {
  	int     ch;
  	char   *p;
-	char	buf[33];
+	char	buf[HEXSIZE];

  	while ((ch = getopt(argc, argv, "pqrs:tx")) != -1)
  		switch (ch) {
@@ -90,7 +94,7 @@

  	if (*argv) {
  		do {
-			p = MD5File(*argv, buf);
+			p = MDFile(*argv, buf);
  			if (!p)
  				warn("%s", *argv);
  			else
@@ -113,15 +117,19 @@
  MDString(const char *string)
  {
  	size_t len = strlen(string);
-	char buf[33];
+	char digest[DIGESTSIZE];
+	char buf[HEXSIZE];

+	Convert2Hex(MD5(string, len, digest), buf);
+
  	if (qflag)
-		printf("%s\n", MD5Data(string, len, buf));
+		printf("%s\n", buf);
  	else if (rflag)
-		printf("%s \"%s\"\n", MD5Data(string, len, buf), string);
+		printf("%s \"%s\"\n", buf, string);
  	else
-		printf("MD5 (\"%s\") = %s\n", string, MD5Data(string, 
len, buf));
+		printf("MD5 (\"%s\") = %s\n", string, buf);
  }
+
  /*
   * Measures the time to digest TEST_BLOCK_COUNT TEST_BLOCK_LEN-byte blocks.
   */
@@ -132,7 +140,8 @@
  	time_t  endTime, startTime;
  	unsigned char block[TEST_BLOCK_LEN];
  	unsigned int i;
-	char   *p, buf[33];
+	unsigned char digest[DIGESTSIZE];
+	char    buf[HEXSIZE];

  	printf
  	    ("MD5 time trial. Digesting %d %d-byte blocks ...",
@@ -147,16 +156,17 @@
  	time(&startTime);

  	/* Digest blocks */
-	MD5Init(&context);
+	MD5_Init(&context);
  	for (i = 0; i < TEST_BLOCK_COUNT; i++)
-		MD5Update(&context, block, TEST_BLOCK_LEN);
-	p = MD5End(&context,buf);
+		MD5_Update(&context, block, TEST_BLOCK_LEN);
+	MD5_Final(digest, &context);
+	Convert2Hex(digest, buf);

  	/* Stop timer */
  	time(&endTime);

  	printf(" done\n");
-	printf("Digest = %s", p);
+	printf("Digest = %s", buf);
  	printf("\nTime = %ld seconds\n", (long) (endTime - startTime));
  	/* Be careful that endTime-startTime is not zero. (Bug fix from Ric
  	 * Anderson, ric@Artisoft.COM.) */
@@ -194,15 +204,63 @@
  	MD5_CTX context;
  	unsigned int len;
  	unsigned char buffer[BUFSIZ];
-	char buf[33];
+	unsigned char digest[DIGESTSIZE];
+	char buf[HEXSIZE];

-	MD5Init(&context);
+	MD5_Init(&context);
  	while ((len = fread(buffer, 1, BUFSIZ, stdin))) {
  		if (tee && len != fwrite(buffer, 1, len, stdout))
  			err(1, "stdout");
-		MD5Update(&context, buffer, len);
+		MD5_Update(&context, buffer, len);
+	}
+	MD5_Final(digest, &context);
+	Convert2Hex(digest, buf);
+
+	printf("%s\n", buf);
+}
+
+/*
+ * Digests the file and returns the hex string result.
+ */
+static char *
+MDFile(char *filename, char *buf)
+{
+	MD5_CTX context;
+	unsigned int len;
+	unsigned char buffer[BUFSIZ];
+	unsigned char digest[DIGESTSIZE];
+	FILE *fp;
+
+	if ((fp = fopen(filename, "r")) == NULL) {
+		return NULL;
+	}
+	MD5_Init(&context);
+	while ((len = fread(buffer, 1, BUFSIZ, fp))) {
+		MD5_Update(&context, buffer, len);
+	}
+
+	MD5_Final(digest, &context);
+	Convert2Hex(digest, buf);
+
+	return buf;
+}
+
+/*
+ * Convert a digest string into an ascii hex string.
+ */
+static char *
+Convert2Hex(unsigned char *digest, char *buf)
+{
+        static const char hex[]="0123456789abcdef";
+	int i;
+
+	for (i = 0; i < DIGESTSIZE; i++) {
+		buf[i+i] = hex[digest[i] >> 4];
+		buf[i+i+1] = hex[digest[i] & 0x0f];
  	}
-	printf("%s\n", MD5End(&context,buf));
+	buf[i+i] = '\0';
+
+	return buf;
  }

  static void

On Mon, Jul 16, 2001 at 10:30:01PM -0700, Dima Dorfman (dima@unixfreak.org) wrote:
>  Why?  What's wrong with:
>  
>  	ln -s /usr/bin/openssl md5
>  
>  works for sha, sha1, ripemd160, and a few other algorithms.  The only
>  thing wrong with it is that openssl doesn't support some of the
>  options md5(1) does, and has a slightly different output format.

The output thing is not a problem -- one can always pipe to
sed(1) to solve it.

-- 
wca

Comment 10 Kris Kennaway 2003-07-13 07:22:45 UTC

State Changed
From-To: open->closed

Thanks for the submission, but as you note openssl already 
has this capability.  Duplicating code would not be a good 
idea.