Bug 233907

Summary: Memory leakage in opencrypto for gcm algorithm
Product: Base System Reporter: Oleg <olegyankovic>
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: Closed FIXED    
Severity: Affects Only Me CC: ae
Priority: ---    
Version: 11.2-RELEASE   
Hardware: amd64   
OS: Any   

Description Oleg 2018-12-10 11:44:26 UTC
Opencrypto subsystem has memory leakage when gcm algorithm is used. To make sure the bug exist it's enough to monitor memory usage through "vmstat -m | grep crypto" while opening and freeing new sessions for gcm (the auth part of gcm: CRYPTO_AES_256_NIST_GMAC).

Memory is allocated at swcr_newsession function:

929 	                case CRYPTO_AES_256_NIST_GMAC:
930 	                        axf = &auth_hash_nist_gmac_aes_256;
931 	                auth4common:
932 	                        len = cri->cri_klen / 8;
933 	                        if (len != 16 && len != 24 && len != 32) {
934 	                                swcr_freesession_locked(dev, i);
935 	                                rw_runlock(&swcr_sessions_lock);
936 	                                return EINVAL;
937 	                        }
938 	
939 	                        (*swd)->sw_ictx = malloc(axf->ctxsize, M_CRYPTO_DATA,
940 	                            M_NOWAIT);

and not released in swcr_freesession_locked function. I think it's enough to make something like this in swcr_freesession_locked:

1055 	                case CRYPTO_MD5:
1056 	                case CRYPTO_SHA1:
                        case CRYPTO_AES_128_NIST_GMAC:
                        case CRYPTO_AES_192_NIST_GMAC:
                        case CRYPTO_AES_256_NIST_GMAC:

1057 	                        axf = swd->sw_axf;
1058 	
1059 	                        if (swd->sw_ictx)
1060 	                                free(swd->sw_ictx, M_CRYPTO_DATA);
1061 	                        break;
Comment 1 commit-hook freebsd_committer freebsd_triage 2018-12-13 09:00:23 UTC
A commit references this bug:

Author: ae
Date: Thu Dec 13 08:59:51 UTC 2018
New revision: 342030
URL: https://svnweb.freebsd.org/changeset/base/342030

Log:
  Plug memory leak for AES_*_NIST_GMAC algorithms.

  swcr_newsession() allocates sw_ictx for these algorithms, thus we need
  to free() it in swcr_freesession().

  PR:		233907
  MFC after:	1 week

Changes:
  head/sys/opencrypto/cryptosoft.c
Comment 2 commit-hook freebsd_committer freebsd_triage 2018-12-20 08:29:16 UTC
A commit references this bug:

Author: ae
Date: Thu Dec 20 08:28:52 UTC 2018
New revision: 342276
URL: https://svnweb.freebsd.org/changeset/base/342276

Log:
  MFC r342030:
    Plug memory leak for AES_*_NIST_GMAC algorithms.

    swcr_newsession() allocates sw_ictx for these algorithms, thus we need
    to free() it in swcr_freesession().

    PR:		233907

Changes:
_U  stable/12/
  stable/12/sys/opencrypto/cryptosoft.c
Comment 3 commit-hook freebsd_committer freebsd_triage 2018-12-20 08:33:24 UTC
A commit references this bug:

Author: ae
Date: Thu Dec 20 08:33:12 UTC 2018
New revision: 342277
URL: https://svnweb.freebsd.org/changeset/base/342277

Log:
  MFC r342030:
    Plug memory leak for AES_*_NIST_GMAC algorithms.

    swcr_newsession() allocates sw_ictx for these algorithms, thus we need
    to free() it in swcr_freesession().

    PR:		233907

Changes:
_U  stable/11/
  stable/11/sys/opencrypto/cryptosoft.c
Comment 4 Andrey V. Elsukov freebsd_committer freebsd_triage 2018-12-20 08:41:30 UTC
Fixed in head/, stable/12 and stable/11. Thanks!