Bug 269979 - net/megacmd: webdav server fail on any version after 1.4.1 (was: v1.5.0 broken / core dump during start)
Summary: net/megacmd: webdav server fail on any version after 1.4.1 (was: v1.5.0 broke...
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-ports-bugs (Nobody)
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2023-03-05 16:31 UTC by Riccardo Torrini
Modified: 2024-06-16 20:23 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Riccardo Torrini 2023-03-05 16:31:03 UTC
Old port v1.4.1 works, after upgrade to v1.5.0 it crash (make a core) during start of mega-cmd-server

(lldb) bt all
* thread #1, name = 'mega-cmd-server', stop reason = signal SIGSEGV
  * frame #0: 0x0000000800e8ce35
    frame #1: 0x0000000800a74448
  thread #2, name = 'mega-cmd-server', stop reason = signal SIGSEGV
    frame #0: 0x00000008017aa1da

I found only this on the same argument:
https://forums.freebsd.org/threads/cannot-get-megacmd-to-run.86644/
(please note we can't use rclone because mega slows it down, I opened a ticket and they convinced me to stop using rclone, the risk of seeing account banned is a good argument, I pay for my backup storage there)

Being unable to fix it myself I pkg-locked previous version ;)

I also noticed this:
-rw-r--r--  1 root  wheel  2032642 Sep 15  2021 meganz-MEGAcmd-1.4.1_Win_GH0.tar.gz
-rw-r--r--  1 root  wheel  2070287 Mar  5 15:56 meganz-MEGAcmd-1.5.0_Linux_GH0.tar.gz

We switched from Win to Linux version (but doubt this is the crash problem source)

Because Makefile has no fancy patches I ingenuously changed distinfo to fetch new v1.5.1, build obviously fail  0= )

Being probably unfixable can we revert to 1.4.1 and/or split pkg to working/not-working flavor?

--
Riccardo.
Comment 1 Fernando Apesteguía freebsd_committer freebsd_triage 2023-03-06 12:57:25 UTC
It looks to me that the m_state attribute is not properly initialized. I'm not familiar with the code in security/cryptocpp. Pinging maintainer to see if he can have more info.

CryptoPP::(anonymous namespace)::SHA256_HashBlock_CXX (state=0x10, data=data@entry=0x7fffffffd090) at sha.cpp:424
424         memcpy(T, state, sizeof(T));
(gdb) p state
$1 = (CryptoPP::word32 *) 0x10
(gdb) p *state
Cannot access memory at address 0x10
Comment 2 kikadf 2023-04-18 16:28:31 UTC
I fix this segmentation fault myself to build megacmd with add -CRYPTOPP_DISABLE_ASM to CXXFLAGS. You can try with the patch in bug #270916.
Comment 3 Riccardo Torrini 2023-04-19 06:24:20 UTC
My two tests:
- rebuild of 1.5.0 with CXX flag: no more crash
- applied your patch, build of 1.6.1, no crash

But :-(

2023-04-19 08:06:48 DEBUG : data/7f/7f...: Received error: HTTP/1.1 500 Internal Server Error
Connection: close: 500 Internal Server Error - low level retry 9/10

Started a sync yesterday night, nothing transferred, only errors.
This evening I will revert to 1.4.1 and check my account.
Anyway, mega-cmd shell seems to work, I can "df", "cd" and "ls" all my stuff, probably only webdav is broken (and/or get/put files)

Thanks a lot for your effort, let me know if you have some ideas about this.

--
Riccardo.
Comment 4 kikadf 2023-04-19 07:17:12 UTC
(In reply to Riccardo Torrini from comment #3)
I have no idea what problem do you have. For me works fine, the folders synced to my local folder, and the new files synced to the cloud. I didn't have older version of megacmd, the 1.6.1 is the first install for me. Where did you see this error messages?
Comment 5 commit-hook freebsd_committer freebsd_triage 2023-04-20 13:36:21 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=c72cc0314bc0e0fe56f220585c57fb09303aea1d

commit c72cc0314bc0e0fe56f220585c57fb09303aea1d
Author:     kikadf <kikadf.01@gmail.com>
AuthorDate: 2023-04-18 15:51:40 +0000
Commit:     Robert Clausecker <fuz@FreeBSD.org>
CommitDate: 2023-04-20 11:49:23 +0000

    net/megacmd: update to 1.6.1

     - update sdk to 4.17.1a
     - fix segmentation fault with cryptopp-8.7.0:
       add -DCRYPTOPP_DISABLE_ASM to CXXFLAGS on i386 and amd64
     - adopt port

    PR:             269979, 270916

 net/megacmd/Makefile  | 13 +++++++++----
 net/megacmd/distinfo  | 10 +++++-----
 net/megacmd/pkg-plist |  4 ++--
 3 files changed, 16 insertions(+), 11 deletions(-)
Comment 6 Riccardo Torrini 2023-04-20 15:15:51 UTC
(In reply to kikadf from comment #4)

I use restic + rclone serve restic (for local compressed/encrypted repository) + megacmd with webdav (to send local copy to mega)

Errors are from rclone (using mega-cmd webdav[1] as backend), when it checks old files works fine, but when it send a new file it receive a 500.

[1] from mega-cmd webdav --help: Caveat: This functionality is in BETA state. If you experience any issue with this, please contact: support@mega.nz


Reinstalled 1.4.1, started again rclone sync using webdav, it works :-(
I'm on 13.1-p3-p3-p5, I can try on 13.2 in the next few days

Transferred:      124.297 MiB / 5.934 GiB, 2%, 4.505 MiB/s, ETA 22m1s
Checks:             16423 / 16423, 100%
Transferred:           51 / 512, 10%
Elapsed time:        51.5s

[...] DEBUG : pacer: Rate limited, increasing sleep to 640ms

2023/04/20 17:07:59 INFO  : 
Transferred:        6.030 GiB / 6.030 GiB, 100%, 4.956 MiB/s, ETA 0s
Checks:             16423 / 16423, 100%
Transferred:          512 / 512, 100%
Elapsed time:     31m28.6s

2023/04/20 17:07:59 DEBUG : 7 go routines active

And yes, mega is really not the best backend for an offsite backup  0= )
Comment 7 Riccardo Torrini 2023-05-27 13:28:36 UTC
(In reply to Riccardo Torrini from comment #6)

updated to 1.6.3, same error using webdav through rclone.
opened a ticket to mega, I'll keep you informed.
Comment 8 Riccardo Torrini 2023-06-11 10:06:48 UTC
(In reply to Riccardo Torrini from comment #7)

mega answer:

Our developers have reviewed your issue and tried to reproduce the issue.
The issue was replicated in 1.5.0 however it works fine in 1.6.3.


Not too bad. But I still have error 500 during sync, runned rclone with --log-level=DEBUG and --dump headers, updated mega ticket with this (here you can see a modified log for readability)

{date} DEBUG : PROPFIND /{webdav_path}/data/52/ HTTP/1.1
{date} DEBUG : HTTP/1.1 207 Multi-Status
{date} DEBUG : MKCOL /{webdav_path}/data/52/ HTTP/1.1
{date} DEBUG : HTTP/1.1 405 Unknown Error
{date} DEBUG : MKCOL /{webdav_path}/data/52/ HTTP/1.1
{date} DEBUG : HTTP/1.1 405 Unknown Error
{date} DEBUG : PUT /{webdav_path}/data/52/{sha256-file_1} HTTP/1.1
{date} DEBUG : PUT /{webdav_path}/data/52/{sha256-file_2} HTTP/1.1
{date} DEBUG : HTTP/1.1 500 Internal Server Error
{date} DEBUG : HTTP/1.1 500 Internal Server Error
{date} DEBUG : DELETE /{webdav_path}/data/52/{sha256-file_1} HTTP/1.1
{date} DEBUG : HTTP/1.1 404 Not Found
{date} DEBUG : {sha256-file_1}: Received error: HTTP/1.1 500 Internal Server Error
{date} DEBUG : MKCOL /{webdav_path}/data/52/ HTTP/1.1
{date} DEBUG : HTTP/1.1 405 Unknown Error
{date} DEBUG : DELETE /{webdav_path}/data/52/{sha256-file_2} HTTP/1.1
{date} DEBUG : HTTP/1.1 404 Not Found
{date} DEBUG : {sha256-file_2}: Received error: HTTP/1.1 500 Internal Server Error
{date} DEBUG : MKCOL /{webdav_path}/data/52/ HTTP/1.1
{date} DEBUG : HTTP/1.1 405 Unknown Error
{date} DEBUG : PUT /{webdav_path}/data/52/{sha256-file_1} HTTP/1.1
{date} DEBUG : HTTP/1.1 500 Internal Server Error
{date} DEBUG : PUT /{webdav_path}/data/52/{sha256-file_2} HTTP/1.1
{date} DEBUG : HTTP/1.1 500 Internal Server Error

(identical config and same rclone version but megacmd 1.4.1 works like a charm)

I don't think that the problem is on my machine but if you have any hint to better debug this issue will be greatly appreciated.
Comment 9 Riccardo Torrini 2023-12-29 14:23:26 UTC
(In reply to Riccardo Torrini from comment #7)

Updated server to 13.2-p8, another test:

Dec 29 12:31:19 hydra pkg[71202]: cryptopp upgraded: 8.5.0_1 -> 8.9.0
Dec 29 12:31:53 hydra pkg[71202]: megacmd upgraded: 1.4.1 -> 1.6.3_2

Same error using webdav, I found this messages during rclone copy
(extracted from megacmdserver.log with a little edit for readability):

# failed using latest version
[API:err: 13:29:07] Failed to open('/home/riccardo/.megaCmd/httputfile.getxfer.3355.1.mega.home/index/{sha256-file_1}'): error 2: No such file or directory
[API:err: 13:29:07] Failed to open('/home/riccardo/.megaCmd/httputfile.getxfer.3355.2.mega.home/index/{sha256-file_2}'): error 2: No such file or directory
[API:err: 13:29:07] Failed to open('/home/riccardo/.megaCmd/httputfile.getxfer.3355.3.mega.home/index/{sha256-file_3}'): error 2: No such file or directory
[API:err: 13:29:07] Failed to open('/home/riccardo/.megaCmd/httputfile.getxfer.3355.4.mega.home/index/{sha256-file_4}'): error 2: No such file or directory

# good with 1.4.1 (found with "ls -lart" inside ~/.megaCmd)
... 16901704 Dec 29 14:54 httputfile.getxfer.3568.326.mega
... 16911823 Dec 29 14:55 httputfile.getxfer.3568.336.mega
... 16922232 Dec 29 14:55 httputfile.getxfer.3568.337.mega
... 16878039 Dec 29 14:55 httputfile.getxfer.3568.338.mega

I think that cache file could be "{fixed-string}.{pid}.{sequence_number}.mega" on 1.4.1 but on new version there is a ".{some_part_of_current_file_name_and_path}" added to the end. I say "some part" because of missing element between "home" and "index", namely {hostname_of_backupped_machine}.
Maybe error 500 is because of missing/mispelled local cached file?

Probably new mega-cmd is badly broken, is not a pkg issue :-(

-- 
Riccardo.
Comment 10 Riccardo Torrini 2023-12-29 22:00:34 UTC
changed title to better track current issue
also sent a detailed bug report to upstream
Comment 11 kikadf 2024-05-24 17:45:20 UTC
megacmd has a new release, 1.7.0. I updated the port, if you can, try that, maybe fixed your problem. bug #279273
Comment 12 Riccardo Torrini 2024-06-16 19:41:28 UTC
(In reply to kikadf from comment #11)

Bad news: webdav still broken :-(


Detailed report:

- applied your patch for 1.7.0 with git apply (only a little issue with lines with spaces)

# fetch -o ~/tmp/megacmd-170.patch 'https://bz-attachments.freebsd.org/attachment.cgi?id=250925'
# cd /usr/ports && git apply ~/tmp/megacmd-170.patch 
/home/riccardo/tmp/megacmd-170.patch:74: trailing whitespace.
/home/riccardo/tmp/megacmd-170.patch:84: trailing whitespace.
/home/riccardo/tmp/megacmd-170.patch:122: trailing whitespace.
/home/riccardo/tmp/megacmd-170.patch:136: trailing whitespace.
/home/riccardo/tmp/megacmd-170.patch:146: trailing whitespace.
warning: squelched 35 whitespace errors
warning: 40 lines add whitespace errors.


- another little error during build:

===>  Configuring for megacmd-1.7.0
aclocal: warning: couldn't open directory 'm4': No such file or directory


Than a lot of compiler warning, but this is an upstream problem.
Removed cryptopp and megacmd, installed 8.9.0 and 1.7.0.
Removed ~/.megaCmd
Started mega-cmd, reconfigured for my account, configured webdav

Same error as in comment #9, previous version 1.4.1 use "/home/riccardo/.megaCmd/httputfile.getxfer.10779.1.mega" for temporary files (that seems to be "{fixed-string}.{pid}.{sequence_number}.mega" imho), this version use "same as before" + ".all after last dot on source" (see below). This is an upstream issue, I have an open ticket with mega...

[API:err: 19:07:50] Failed to open('/home/riccardo/.megaCmd/httputfile.getxfer.10779.1.mega.home/data/52/527ede416be5014d641c6c3d404b16406b2dabe73bdfffc6f49c0204362baf0d'): error 2: No such file or directory
Comment 13 Riccardo Torrini 2024-06-16 20:23:39 UTC
This is (part of) one of my trouble tickets to mega:

----- 8< -----

I'm not fluent on C++ programming but I think that you can investigate megaapi_impl.cpp, inside MegaHTTPServer::onBody()

(diff between sdk-3.9.11a and sdk-4.17.1d, as an example)
----- 8< -----
if (parser->method == HTTP_PUT)
{
//create tmp file with contents in messageBody
if (!httpctx->tmpFileAccess)
{
httpctx->tmpFileName=httpctx->server->basePath;
httpctx->tmpFileName.append("httputfile");
- LocalPath suffix;
- httpctx->server->fsAccess->tmpnamelocal(suffix);
- httpctx->tmpFileName.append(suffix.toPath(*httpctx->server->fsAccess));
+ httpctx->tmpFileName.append(LocalPath::tmpNameLocal().toPath(false));

string ext;
- LocalPath localpath = LocalPath::fromPath(httpctx->path, *httpctx->server->fsAccess);
+ LocalPath localpath = LocalPath::fromAbsolutePath(httpctx->path);
if (httpctx->server->fsAccess->getextension(localpath, ext))
{
httpctx->tmpFileName.append(ext);
}
----- 8< -----

Could the problem be triggered by files without an extension?

Unfortunately (for me at least) last answer was:

Our developers are chacking the issue but they have answered that we won't be fixing that any time soon though. We will contact you again if we solve this, we are sorry for the inconvenience.