I tried setting up a Mono application in a iocage base jail (iocage -b ...) but noticed that my certificate store was empty, and that I couldn't use the cert-sync tool to sync with my system certificate store - the tool threw error messages that my file system was mounted as read only. I setup an nearly identical new jail, but made it a 'thick' jail instead (iocage -T ...) instead, and now my Mono certificate store was full of CAs (as expected). Running cert-sync again worked fine (although it was already in sync). So I suspect that cert-sync (and mozroots) tries to write to a folder that is mounted as read-only in a iocage base jail. This is a list of read-only folders on one such jail: /bin /boot /lib /libexec /rescue /sbin /usr/bin /usr/include /usr/lib /usr/libexec /usr/sbin /usr/share /usr/libdata /usr/lib32 I couldn't make either mozroots or cert-sync to be more verbose (as to which folder it was trying to write), but I _guess_ that it's /usr/lib or so. An acceptable workaround is to use a thick jail instead, but it would be _nice_ to have it working in a base jail as well.
I'm experiencing this as well, although with my hand-rolled thinjails. It looks like cert-sync tries to write to /usr/share/.mono, which is the culprit here. Running cert-sync yields the following: terra|worr|22:47:17|1$ sudo jexec j /usr/local/bin/cert-sync /usr/local/etc/ssl/cert.pem Mono Certificate Store Sync - version 6.8.0.123 Populate Mono certificate store from a concatenated list of certificates. Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed. Importing into legacy system store: I already trust 0, your new list has 130 Warning: Could not import C=ES, O=FNMT-RCM, OU=AC RAIZ FNMT-RCM System.IO.IOException: Read-only file system at System.IO.FileSystem.CreateDirectory (System.String fullPath) [0x00191] in <0e6cb1433c7b46f598f86593dd03f528>:0 at System.IO.Directory.CreateDirectory (System.String path) [0x0002c] in <0e6cb1433c7b46f598f86593dd03f528>:0 at Mono.Security.X509.X509Store.CheckStore (System.String path, System.Boolean throwException) [0x00020] in <9d0b4d46cb9c4cd288c22cd9cdf5212a>:0 at Mono.Security.X509.X509Store.Import (Mono.Security.X509.X509Certificate certificate) [0x00000] in <9d0b4d46cb9c4cd288c22cd9cdf5212a>:0 at Mono.Tools.CertSync.ImportToStore (Mono.Security.X509.X509CertificateCollection roots, Mono.Security.X509.X509Store store) [0x00050] in <34bb119f69354d8986322c88a4400682>:0 Warning: Could not import C=ES, O=FNMT-RCM, OU=Ceres, OID.2.5.4.97=VATES-Q2826004J, CN=AC RAIZ FNMT-RCM SERVIDORES SEGUROS ... Running the following dtrace script yields: terra|worr|23:08:39|130$ cat ro-cert-sync.d #!/usr/sbin/dtrace -s syscall:freebsd:mkdir:entry { self->path = copyinstr(arg0); } syscall:freebsd:mkdir:return { if (args[0] != 0) { printf("Could not create %s: %d", self->path, errno); } } terra|worr|23:08:41|0$ sudo dtrace -s ro-cert-sync.d -c 'jexec j /usr/local/bin/cert-sync --quiet /usr/local/etc/ssl/cert.pem' | head -30 dtrace: script 'ro-cert-sync.d' matched 5 probes dtrace: pid 35449 has exited CPU ID FUNCTION:NAME 6 77594 mkdir:return Could not create /usr/share/.mono: 30 6 77594 mkdir:return Could not create /usr/share/.mono/certs: 2 6 77594 mkdir:return Could not create /usr/share/.mono/certs/Trust: 2 6 77594 mkdir:return Could not create /usr/share/.mono: 30 6 77594 mkdir:return Could not create /usr/share/.mono/certs: 2 6 77594 mkdir:return Could not create /usr/share/.mono/certs/Trust: 2 6 77594 mkdir:return Could not create /usr/share/.mono: 30 6 77594 mkdir:return Could not create /usr/share/.mono/certs: 2 6 77594 mkdir:return Could not create /usr/share/.mono/certs/Trust: 2 6 77594 mkdir:return Could not create /usr/share/.mono: 30 6 77594 mkdir:return Could not create /usr/share/.mono/certs: 2 6 77594 mkdir:return Could not create /usr/share/.mono/certs/Trust: 2 6 77594 mkdir:return Could not create /usr/share/.mono: 30 6 77594 mkdir:return Could not create /usr/share/.mono/certs: 2 6 77594 mkdir:return Could not create /usr/share/.mono/certs/Trust: 2 6 77594 mkdir:return Could not create /usr/share/.mono: 30 6 77594 mkdir:return Could not create /usr/share/.mono/certs: 2 6 77594 mkdir:return Could not create /usr/share/.mono/certs/Trust: 2 6 77594 mkdir:return Could not create /usr/share/.mono: 30 6 77594 mkdir:return Could not create /usr/share/.mono/certs: 2 6 77594 mkdir:return Could not create /usr/share/.mono/certs/Trust: 2 6 77594 mkdir:return Could not create /usr/share/.mono: 30 6 77594 mkdir:return Could not create /usr/share/.mono/certs: 2 6 77594 mkdir:return Could not create /usr/share/.mono/certs/Trust: 2 6 77594 mkdir:return Could not create /usr/share/.mono: 30 6 77594 mkdir:return Could not create /usr/share/.mono/certs: 2 6 77594 mkdir:return Could not create /usr/share/.mono/certs/Trust: 2 6 77594 mkdir:return Could not create /usr/share/.mono: 30 6 77594 mkdir:return Could not create /usr/share/.mono/certs: 2
I am also seeing this issue running cert-sync inside of a TrueNAS 12.2 Jail. I am running a number of programmes dependant on mono, all of which are currently non functional with 6.8.0.123 as I cannot get them get mono the certs it needs. Cert-sync —-user runs without issue, but it seems I need to run it machine wide as this doesn’t solve my issue.
We are having the same issue here. Can also confirm that certificates can be imported using the --user flag however this doesn't solve the issue.
I have identified where the problem is. Basically, certificates are stored in the folder Environment.SpecialFolder.CommonApplicationData/.mono/certs as per [0]. However the path of Environment.SpecialFolder.CommonApplicationData is set to "/usr/share" (linuxism). The mono5.10 port includes the proper patches but for some reason they haven't been carried over to the mono6.8 port. The issue likely didn't appear earlier as the certificates shipped by upstream were still valid. You can fix the issue by adding the below two commands in the post-patch section of the Makefile of the port: post-patch: ${REINPLACE_CMD} -e 's|/usr/share|${PREFIX}|g' \ ${WRKSRC}/mcs/class/corlib/System/Environment.cs \ ${WRKSRC}/external/corefx/src/System.Runtime.Extensions/src/System/Environment.Unix.cs ${REINPLACE_CMD} -e 's|/usr/share/.mono|${PREFIX}/share/mono|g' \ ${WRKSRC}/man/mono-configuration-crypto.1 \ ${WRKSRC}/man/mono.1 \ ${WRKSRC}/man/mozroots.1 I am not sure why these changes have been implemented as post-patches instead of regular patch files, the maintainer may simply have run out of time at the time. The first command replaces "/usr/share" with /usr/local/ in the source files mcs/class/corlib/System.Environment.cs and external/corefx/src/System.Runtime.Extensions/src/System.Environment.Unix. The second command replaces /usr/share/.mono with /usr/local/share/mono for a couple of man files. With regards to replacing "/usr/share/.mono" with "/usr/local/mono" there is also a need to copy over the following patch files from the 5.10 ports: - patch-mcs_class_Mono.Security_Mono.Security.Cryptography_KeyPairPersistence.cs - patch-mcs_class_Mono.Security_Mono.Security.X509_X509StoreManager.cs - patch-mcs_tools_mono-configuration-crypto_lib_Mono.Configuration.Crypto_KeyContainerCollection.cs These files directly relate to the certificate manager so applying these patches is important. Then build from source and cert-sync should work as expected. At do not have access to a proper FreeBSD lab at the minute to implement it and propose clean files, but wanted to throw this here in the meantime in case this helps someone being stuck.
[0]: https://github.com/mono/mono/blob/8266c5604b8c03882f2b06af27fdea46b142d6b9/mcs/class/Mono.Security/Mono.Security.X509/X509StoreManager.cs#L44 [1]: https://github.com/mono/mono/blob/bdd772531d379b4e78593587d15113c37edd4a64/mcs/class/corlib/System/Environment.cs#L743
I have just built mono with these changes and while they fix the issue with cert-sync, the CERTIFICATE_VERIFY_FAIL exception still is there, maybe an additional patch needs to be included: csharp> var wc = new System.Net.WebClient(); csharp> wc.DownloadString("https://letsencrypt.org") System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED at /usr/local/ports/lang/mono6.8/work/mono-6.8.0.123/external/boringssl/ssl/handshake_client.c:1132 at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool) at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <6e75d52b61f74f63805d7d058bef4ea1>:0 --- End of inner exception stack trace --- at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <6e75d52b61f74f63805d7d058bef4ea1>:0 --- End of inner exception stack trace --- at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at System.Net.WebOperation.Run () [0x0009a] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at System.Net.HttpWebRequest.GetResponse () [0x00016] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at System.Net.WebClient.DownloadBits (System.Net.WebRequest request, System.IO.Stream writeStream) [0x000e6] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at System.Net.WebClient.DownloadDataInternal (System.Uri address, System.Net.WebRequest& request) [0x00061] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at System.Net.WebClient.DownloadString (System.Uri address) [0x00011] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at System.Net.WebClient.DownloadString (System.String address) [0x00008] in <6e75d52b61f74f63805d7d058bef4ea1>:0 at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadString(string) at <InteractiveExpressionClass>.Host (System.Object& $retval) [0x00000] in <25d4ed1848e74873a46a27201620fdc9>:0 at Mono.CSharp.Evaluator.Evaluate (System.String input, System.Object& result, System.Boolean& result_set) [0x00038] in <50208ddc2c9e4fcea9d1930b33f7702a>:0 at Mono.CSharpShell.Evaluate (System.String input) [0x00000] in <c8753e93298d4958b29320a00cdcfd88>:0
So actually this let's encrypt problem is caused by a separate issue (you must ensure that the DST Root certificate is removed from your certificate because of a bug in a openssl). Therefore the fix suggested fully fixes the issue. I will send the updated Makefile along with the patches to include later today.
Is there any progress on providing a fix in the ports collection? The problem seems to persist, I'm trying it now with the manual fix provided but I think it would still be valuable to be fixed in general.
(In reply to freebsd from comment #9) Yes please somebody help. I blew away my jail running Jackett to move it from one host to another and now this issue is preventing me from getting Jackett back up and running. I'm far to much of a newb to implement the fix discussed earlier, i wouldn't know how to do this. If someone can help with some steps to implement the fix i'll give it a try but needs to be in newbie language :) Thanks.
Is there any progress being made with this issue? I'm counting the days until Sonarr switches to .NET and I don't have to fallback to 6.8.0.105 on every update. Is there some kind of walk-though I can follow to fix this that doesn't require building anything from source? Thanks.
My workaround is to use a normal jail, and not as a base jail/similar.
(In reply to Oscar Carlsson from comment #12) Thanks a million works like a charm.
^Triage: reset Assignee after ports 4553130a9c53.
^Triage: reset Assignee after ports c59052ead113. (ignore previous comment, it referenced bogus commit)
FWIW Patching mono's runtime appears to be the solution the Synology community took when addressing this issues with respect to Sonarr. https://github.com/SynoCommunity/spksrc/pull/5604mono6.8