Bug 258709 - lang/mono6.8: cert-sync doesn't work on iocage style base jails
Summary: lang/mono6.8: cert-sync doesn't work on iocage style base jails
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-ports-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-09-24 19:44 UTC by Oscar Carlsson
Modified: 2023-09-04 02:48 UTC (History)
11 users (show)

See Also:
bugzilla: maintainer-feedback? (mono)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Oscar Carlsson 2021-09-24 19:44:44 UTC

    
Comment 1 Oscar Carlsson 2021-09-24 19:46:02 UTC
I tried setting up a Mono application in a iocage base jail (iocage -b ...) but noticed that my certificate store was empty, and that I couldn't use the cert-sync tool to sync with my system certificate store - the tool threw error messages that my file system was mounted as read only.

I setup an nearly identical new jail, but made it a 'thick' jail instead (iocage -T ...) instead, and now my Mono certificate store was full of CAs (as expected). Running cert-sync again worked fine (although it was already in sync).

So I suspect that cert-sync (and mozroots) tries to write to a folder that is mounted as read-only in a iocage base jail.

This is a list of read-only folders on one such jail:

/bin
/boot
/lib
/libexec
/rescue
/sbin
/usr/bin
/usr/include
/usr/lib
/usr/libexec
/usr/sbin
/usr/share
/usr/libdata
/usr/lib32

I couldn't make either mozroots or cert-sync to be more verbose (as to which folder it was trying to write), but I _guess_ that it's /usr/lib or so.

An acceptable workaround is to use a thick jail instead, but it would be _nice_ to have it working in a base jail as well.
Comment 2 will 2021-10-11 21:09:39 UTC
I'm experiencing this as well, although with my hand-rolled thinjails. It looks like cert-sync tries to write to /usr/share/.mono, which is the culprit here.

Running cert-sync yields the following:

terra|worr|22:47:17|1$ sudo jexec j /usr/local/bin/cert-sync /usr/local/etc/ssl/cert.pem
Mono Certificate Store Sync - version 6.8.0.123
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 0, your new list has 130
Warning: Could not import C=ES, O=FNMT-RCM, OU=AC RAIZ FNMT-RCM
System.IO.IOException: Read-only file system
  at System.IO.FileSystem.CreateDirectory (System.String fullPath) [0x00191] in <0e6cb1433c7b46f598f86593dd03f528>:0 
  at System.IO.Directory.CreateDirectory (System.String path) [0x0002c] in <0e6cb1433c7b46f598f86593dd03f528>:0 
  at Mono.Security.X509.X509Store.CheckStore (System.String path, System.Boolean throwException) [0x00020] in <9d0b4d46cb9c4cd288c22cd9cdf5212a>:0 
  at Mono.Security.X509.X509Store.Import (Mono.Security.X509.X509Certificate certificate) [0x00000] in <9d0b4d46cb9c4cd288c22cd9cdf5212a>:0 
  at Mono.Tools.CertSync.ImportToStore (Mono.Security.X509.X509CertificateCollection roots, Mono.Security.X509.X509Store store) [0x00050] in <34bb119f69354d8986322c88a4400682>:0 
Warning: Could not import C=ES, O=FNMT-RCM, OU=Ceres, OID.2.5.4.97=VATES-Q2826004J, CN=AC RAIZ FNMT-RCM SERVIDORES SEGUROS
...

Running the following dtrace script yields:

terra|worr|23:08:39|130$ cat ro-cert-sync.d 
#!/usr/sbin/dtrace -s

syscall:freebsd:mkdir:entry {
  self->path = copyinstr(arg0);
}

syscall:freebsd:mkdir:return {
  if (args[0] != 0) {
    printf("Could not create %s: %d", self->path, errno);
  }
}
terra|worr|23:08:41|0$ sudo dtrace -s ro-cert-sync.d -c 'jexec j /usr/local/bin/cert-sync --quiet /usr/local/etc/ssl/cert.pem' | head -30
dtrace: script 'ro-cert-sync.d' matched 5 probes
dtrace: pid 35449 has exited
CPU     ID                    FUNCTION:NAME
  6  77594                     mkdir:return Could not create /usr/share/.mono: 30
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs/Trust: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono: 30
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs/Trust: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono: 30
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs/Trust: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono: 30
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs/Trust: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono: 30
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs/Trust: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono: 30
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs/Trust: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono: 30
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs/Trust: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono: 30
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs/Trust: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono: 30
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs/Trust: 2
  6  77594                     mkdir:return Could not create /usr/share/.mono: 30
  6  77594                     mkdir:return Could not create /usr/share/.mono/certs: 2
Comment 3 Jake Bennett-Young 2021-10-17 11:01:24 UTC
I am also seeing this issue running cert-sync inside of a TrueNAS 12.2 Jail.

I am running a number of programmes dependant on mono, all of which are currently non functional with 6.8.0.123 as I cannot get them get mono the certs it needs.

Cert-sync —-user runs without issue, but it seems I need to run it machine wide as this doesn’t solve my issue.
Comment 4 Misso Works 2021-10-18 15:34:12 UTC
We are having the same issue here. Can also confirm that certificates can be imported using the --user flag however this doesn't solve the issue.
Comment 5 Misso Works 2021-10-18 16:27:56 UTC
I have identified where the problem is. Basically, certificates are stored in the folder Environment.SpecialFolder.CommonApplicationData/.mono/certs as per [0]. However the path of Environment.SpecialFolder.CommonApplicationData is set to "/usr/share" (linuxism).

The mono5.10 port includes the proper patches but for some reason they haven't been carried over to the mono6.8 port. The issue likely didn't appear earlier as the certificates shipped by upstream were still valid. You can fix the issue by adding the below two commands in the post-patch section of the Makefile of the port:

post-patch:
        ${REINPLACE_CMD} -e 's|/usr/share|${PREFIX}|g' \
		${WRKSRC}/mcs/class/corlib/System/Environment.cs \
		${WRKSRC}/external/corefx/src/System.Runtime.Extensions/src/System/Environment.Unix.cs
	${REINPLACE_CMD} -e 's|/usr/share/.mono|${PREFIX}/share/mono|g' \
		${WRKSRC}/man/mono-configuration-crypto.1 \
		${WRKSRC}/man/mono.1 \
		${WRKSRC}/man/mozroots.1

I am not sure why these changes have been implemented as post-patches instead of regular patch files, the maintainer may simply have run out of time at the time.

The first command replaces "/usr/share" with /usr/local/ in the source files mcs/class/corlib/System.Environment.cs and external/corefx/src/System.Runtime.Extensions/src/System.Environment.Unix.

The second command replaces /usr/share/.mono with /usr/local/share/mono for a couple of man files. With regards to replacing "/usr/share/.mono" with "/usr/local/mono" there is also a need to copy over the following patch files from the 5.10 ports:

- patch-mcs_class_Mono.Security_Mono.Security.Cryptography_KeyPairPersistence.cs	- patch-mcs_class_Mono.Security_Mono.Security.X509_X509StoreManager.cs
- patch-mcs_tools_mono-configuration-crypto_lib_Mono.Configuration.Crypto_KeyContainerCollection.cs	

These files directly relate to the certificate manager so applying these patches is important. Then build from source and cert-sync should work as expected. At do not have access to a proper FreeBSD lab at the minute to implement it and propose clean files, but wanted to throw this here in the meantime in case this helps someone being stuck.
Comment 7 Misso Works 2021-10-19 12:29:17 UTC
I have just built mono with these changes and while they fix the issue with cert-sync, the CERTIFICATE_VERIFY_FAIL exception still is there, maybe an additional patch needs to be included:

csharp> var wc = new System.Net.WebClient();
csharp> wc.DownloadString("https://letsencrypt.org")                                                                                                                              System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED                                                      at /usr/local/ports/lang/mono6.8/work/mono-6.8.0.123/external/boringssl/ssl/handshake_client.c:1132                                                                               at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <6e75d52b61f74f63805d7d058bef4ea1>:0                                                                                at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <6e75d52b61f74f63805d7d058bef4ea1>:0                                                                                                                                                                       at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)                                 at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <6e75d52b61f74f63805d7d058bef4ea1>:0                                
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
   --- End of inner exception stack trace ---
  at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at System.Net.WebOperation.Run () [0x0009a] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00016] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at System.Net.WebClient.DownloadBits (System.Net.WebRequest request, System.IO.Stream writeStream) [0x000e6] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at System.Net.WebClient.DownloadDataInternal (System.Uri address, System.Net.WebRequest& request) [0x00061] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at System.Net.WebClient.DownloadString (System.Uri address) [0x00011] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at System.Net.WebClient.DownloadString (System.String address) [0x00008] in <6e75d52b61f74f63805d7d058bef4ea1>:0 
  at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadString(string)
  at <InteractiveExpressionClass>.Host (System.Object& $retval) [0x00000] in <25d4ed1848e74873a46a27201620fdc9>:0 
  at Mono.CSharp.Evaluator.Evaluate (System.String input, System.Object& result, System.Boolean& result_set) [0x00038] in <50208ddc2c9e4fcea9d1930b33f7702a>:0 
  at Mono.CSharpShell.Evaluate (System.String input) [0x00000] in <c8753e93298d4958b29320a00cdcfd88>:0
Comment 8 Misso Works 2021-10-20 06:47:19 UTC
So actually this let's encrypt problem is caused by a separate issue (you must ensure that the DST Root certificate is removed from your certificate because of a bug in a openssl). Therefore the fix suggested fully fixes the issue. I will send the updated Makefile along with the patches to include later today.
Comment 9 freebsd 2022-06-20 18:16:54 UTC
Is there any progress on providing a fix in the ports collection? The problem seems to persist, I'm trying it now with the manual fix provided but I think it would still be valuable to be fixed in general.
Comment 10 LFK 2022-06-29 07:40:11 UTC
(In reply to freebsd from comment #9)
Yes please somebody help.  I blew away my jail running Jackett to move it from one host to another and now this issue is preventing me from getting Jackett back up and running.  I'm far to much of a newb to implement the fix discussed earlier, i wouldn't know how to do this.

If someone can help with some steps to implement the fix i'll give it a try but needs to be in newbie language :)

Thanks.
Comment 11 jaskerx 2022-08-15 17:00:07 UTC
Is there any progress being made with this issue? I'm counting the days until Sonarr switches to .NET and I don't have to fallback to 6.8.0.105 on every update. Is there some kind of walk-though I can follow to fix this that doesn't require building anything from source?

Thanks.
Comment 12 Oscar Carlsson 2022-08-16 04:45:39 UTC
My workaround is to use a normal jail, and not as a base jail/similar.
Comment 13 jaskerx 2022-08-16 23:29:40 UTC
(In reply to Oscar Carlsson from comment #12)

Thanks a million works like a charm.
Comment 14 Jan Beich freebsd_committer freebsd_triage 2023-08-10 11:15:44 UTC
^Triage: reset Assignee after ports 4553130a9c53.
Comment 15 Jan Beich freebsd_committer freebsd_triage 2023-08-10 11:18:24 UTC
^Triage: reset Assignee after ports c59052ead113. (ignore previous comment, it referenced bogus commit)
Comment 16 Daniel Kimsey 2023-09-04 02:48:25 UTC
FWIW Patching mono's runtime appears to be the solution the Synology community took when addressing this issues with respect to Sonarr.

https://github.com/SynoCommunity/spksrc/pull/5604mono6.8