Bug 224580 - [deskutils/calibre]: MTP devices not detected
Summary: [deskutils/calibre]: MTP devices not detected
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Guido Falsi
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-12-25 06:19 UTC by Conrad Meyer
Modified: 2018-01-06 16:29 UTC (History)
1 user (show)

See Also:
madpilot: maintainer-feedback+


Attachments
Patch including suggested changes (5.27 KB, patch)
2017-12-26 10:59 UTC, Guido Falsi
no flags Details | Diff
Update to version including upstreamed changes (3.51 KB, patch)
2018-01-05 14:02 UTC, Guido Falsi
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Conrad Meyer freebsd_committer 2017-12-25 06:19:25 UTC
calibre, version 3.14.0
ERROR: Debugging failed: Running debug device detection failed. Click Show Details for more information.

"No plugin named u'libmtp'"

Traceback (most recent call last):
  File "/usr/local/lib/calibre/calibre/gui2/device.py", line 87, in run
    self.result = self.func(*self.args, **self.kwargs)
  File "/usr/local/lib/calibre/calibre/gui2/device.py", line 459, in _debug_detection
    disabled_plugins=self.disabled_device_plugins)
  File "/usr/local/lib/calibre/calibre/devices/__init__.py", line 129, in debug
    if dev.debug_managed_device_detection(s.devices, buf):
  File "/usr/local/lib/calibre/calibre/devices/mtp/base.py", line 26, in synchronizer
    return func(self, *args, **kwargs)
  File "/usr/local/lib/calibre/calibre/devices/mtp/unix/driver.py", line 133, in debug_managed_device_detection
    err = plugins['libmtp'][1]
  File "/usr/local/lib/calibre/calibre/constants.py", line 207, in __getitem__
    raise KeyError('No plugin named %r'%name)
KeyError: "No plugin named u'libmtp'"
Comment 1 Conrad Meyer freebsd_committer 2017-12-25 06:31:58 UTC
Not sure if related.

http://beefy9.nyi.freebsd.org/data/111amd64-default/456877/logs/calibre-3.14.0.log :

===>   calibre-3.14.0 depends on shared library: libmtp.so - not found
===>   Installing existing package /packages/All/libmtp-1.1.14.txz
[111amd64-default-job-16] Installing libmtp-1.1.14...
[111amd64-default-job-16] Extracting libmtp-1.1.14: .......... done
===>   calibre-3.14.0 depends on shared library: libmtp.so - found (/usr/local/lib/libmtp.so)
Comment 2 Guido Falsi freebsd_committer 2017-12-25 09:33:10 UTC
(In reply to Conrad Meyer from comment #0)
> calibre, version 3.14.0
> ERROR: Debugging failed: Running debug device detection failed. Click Show
> Details for more information.

What were you doing when this error happened?

> 
> "No plugin named u'libmtp'"

libmtp is actually registered as a LIB_DEPENDS in the port, so it's installed at runtime together with it.

The problem looks a little more complex.

Just for a test have you tried forcing reinstallation of the libmtp and calibre ports?
Comment 3 Conrad Meyer freebsd_committer 2017-12-25 10:00:42 UTC
I'm trying to sync books onto an MTP device (Nook Glowlight Plus, which is basically Android).

I was confused by the libmtp name.  I see that the port depends on that external library.

I think the source of the confusion is that calibre *also* has some internal plugin also called libmtp, which is what is missing.  On my Linux system (which can successfully talk to the device), the calibre package has /usr/lib64/calibre/calibre/plugins/libmtp.so.  But the FreeBSD port somehow doesn't have that plugin.
Comment 4 Conrad Meyer freebsd_committer 2017-12-25 10:05:55 UTC
The internal libmtp appears to be a thin Python wrapper around C libmtp: https://github.com/kovidgoyal/calibre/blob/master/src/calibre/devices/mtp/unix/libmtp.c

It appears to not be built outside of OS X, Linux, and (?!)Haiku: https://github.com/kovidgoyal/calibre/blob/master/setup/extensions.json#L197

Probably that last one should include FreeBSD.  Possibly some of those other Linux-inclusive lines that do not include FreeBSD as well.
Comment 5 Conrad Meyer freebsd_committer 2017-12-25 10:08:42 UTC
Looks like the string we want is "bsd": https://github.com/kovidgoyal/calibre/blob/master/setup/build.py#L91
Comment 6 Guido Falsi freebsd_committer 2017-12-25 17:49:59 UTC
(In reply to Conrad Meyer from comment #5)
> Looks like the string we want is "bsd":
> https://github.com/kovidgoyal/calibre/blob/master/setup/build.py#L91

Definitely, Thanks for your analysis!

The MTP protocol is used only with certain devices, I don't have any using such protocol, so I could never test it.

I'll prepare a patch, which I'll also propose upstream.
Comment 7 Conrad Meyer freebsd_committer 2017-12-25 17:56:16 UTC
(In reply to Guido Falsi from comment #6)
> The MTP protocol is used only with certain devices,

It's common to recentish Android devices — phones, tablets.

> I don't have any using such protocol, so I could never test it.

No worries — I have an iphone and this is my first MTP e-reader too.  Thanks for maintaining calibre!
Comment 8 Conrad Meyer freebsd_committer 2017-12-25 18:14:48 UTC
By the way, the port seems to be missing a build-time dependency on hunspell.  I'm not sure how it builds without it.  Maybe it is a new dependency in the git master branch.
Comment 9 Conrad Meyer freebsd_committer 2017-12-25 18:25:07 UTC
I've submitted this change upstream, as well as one of our local patches: https://github.com/kovidgoyal/calibre/pull/795
Comment 10 Guido Falsi freebsd_committer 2017-12-25 19:49:40 UTC
(In reply to Conrad Meyer from comment #8)
> By the way, the port seems to be missing a build-time dependency on
> hunspell.  I'm not sure how it builds without it.  Maybe it is a new
> dependency in the git master branch.

There is a bundled version of hunspell included in the calibre sources. It's actually quite old, so maybe it's worth investigating using an external one, if that does not create problems.
Comment 11 Guido Falsi freebsd_committer 2017-12-25 20:01:11 UTC
(In reply to Conrad Meyer from comment #9)
> I've submitted this change upstream, as well as one of our local patches:
> https://github.com/kovidgoyal/calibre/pull/795

Thank you.

You actually created a patch including things I have been too lazy to do/upstream. Thank you a lot!

I'm integrating that all in the port and testing it. I'll followup with a patch for the port soon.
Comment 12 Guido Falsi freebsd_committer 2017-12-25 20:08:09 UTC
(In reply to Guido Falsi from comment #10)
> (In reply to Conrad Meyer from comment #8)
> > By the way, the port seems to be missing a build-time dependency on
> > hunspell.  I'm not sure how it builds without it.  Maybe it is a new
> > dependency in the git master branch.
> 
> There is a bundled version of hunspell included in the calibre sources. It's
> actually quite old, so maybe it's worth investigating using an external one,
> if that does not create problems.

Now that I give it a better look, it looks like the bundled(and customised) hunspell IS the hunspell module, compiled without external dependencies, nor a way to link to an external one, so I don't see any option but to leave it as is.

In fact I also have memory of having already gone through this reasoning in the past.
Comment 13 Guido Falsi freebsd_committer 2017-12-26 10:59:23 UTC
Created attachment 189110 [details]
Patch including suggested changes

I created and tested this patch including the changes you suggest.

I also noticed a pair of scripts which did not get their shebangs fixed and corrected that too.

Could you test it and report back if it really works as expected and solves the problem you've been experiencing?

Thanks in advance.
Comment 14 Conrad Meyer freebsd_committer 2017-12-26 19:29:41 UTC
(In reply to Guido Falsi from comment #10)
(In reply to Guido Falsi from comment #12)
At least with git master, I couldn't compile calibre without installing the port hunspell package.

(In reply to Guido Falsi from comment #13)
It builds fine.  As far as running it, my FreeBSD machine I can build ports on is headless and I'm having some trouble testing with it.  Currently Qt refuses to start (maybe) because of something to do with xkb:

xkbcommon: ERROR: failed to add default include path /usr/local/share/X11/xkb
Qt: Failed to create XKB context!
Use QT_XKB_CONFIG_ROOT environmental variable to provide an additional search path, add ':' as separator to provide several search paths and/or make sure that XKB configuration data directory contains recent enough contents, to update please see http://cgit.freedesktop.org/xkeyboard-config/ .
process 68929: D-Bus library appears to be incorrectly set up; failed to read machine uuid: Failed to open "/etc/machine-id": No such file or directory
See the manual page for dbus-uuidgen to correct this issue.
  D-Bus not built with -rdynamic so unable to print a backtrace
[1]    68929 abort (core dumped)  calibre
Comment 15 Conrad Meyer freebsd_committer 2017-12-26 19:41:44 UTC
(In reply to Conrad Meyer from comment #14)
> xkbcommon: ERROR: failed to add default include path /usr/local/share/X11/xkb
> Qt: Failed to create XKB context!
> Use QT_XKB_CONFIG_ROOT environmental variable to provide an additional search
> path, add ':' as separator to provide several search paths and/or make sure
> that XKB configuration data directory contains recent enough contents, to
> update please see http://cgit.freedesktop.org/xkeyboard-config/ .

This was a (?)missing runtime dependency on xkeyboard-common, probably missing from Qt.

> process 68929: D-Bus library appears to be incorrectly set up; failed to read
> machine uuid: Failed to open "/etc/machine-id": No such file or directory
> See the manual page for dbus-uuidgen to correct this issue.
>  D-Bus not built with -rdynamic so unable to print a backtrace

This is a broken dbus package not initializing /etc/machine-id on installation.

Next issue:
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
(crash)

bt:
(gdb) bt
#0  0x000000081f3985ee in ?? () from /usr/local/lib/qt5/plugins/xcbglintegrations/libqxcb-glx-integration.so
#1  0x000000081c76660a in QXcbWindow::create() () from /usr/local/lib/qt5/libQt5XcbQpa.so.5
#2  0x000000081c758a42 in QXcbIntegration::createPlatformWindow(QWindow*) const () from /usr/local/lib/qt5/libQt5XcbQpa.so.5
#3  0x000000080c706401 in QWindowPrivate::create(bool) () from /usr/local/lib/qt5/libQt5Gui.so.5
#4  0x000000080bd9c883 in QWidgetPrivate::create_sys(unsigned long long, bool, bool) () from /usr/local/lib/qt5/libQt5Widgets.so.5
#5  0x000000080bd9b596 in QWidget::create(unsigned long long, bool, bool) () from /usr/local/lib/qt5/libQt5Widgets.so.5
#6  0x000000080bdad05f in QWidget::setVisible(bool) () from /usr/local/lib/qt5/libQt5Widgets.so.5
#7  0x000000080bf4d90d in QDialog::setVisible(bool) () from /usr/local/lib/qt5/libQt5Widgets.so.5
#8  0x000000080b2bd42c in sipQWizard::setVisible(bool) () from /usr/local/lib/python2.7/site-packages/PyQt5/QtWidgets.so
#9  0x000000080bf4d08a in QDialog::exec() () from /usr/local/lib/qt5/libQt5Widgets.so.5
#10 0x000000080b4b65cb in meth_QDialog_exec_(_object*, _object*) () from /usr/local/lib/python2.7/site-packages/PyQt5/QtWidgets.so
#11 0x0000000800b34d05 in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.7.so.1
#12 0x0000000800b31ed5 in PyEval_EvalCodeEx () from /usr/local/lib/libpython2.7.so.1

:-/
Comment 16 Conrad Meyer freebsd_committer 2017-12-26 20:24:39 UTC
Hm, when I install the patched pkg on my laptop, I still get the same backtrace on startup (KeyError: "No plugin named u'libmtp'").
Comment 17 Conrad Meyer freebsd_committer 2017-12-26 20:26:54 UTC
Here's the problem:

constants.py:

class Plugins(collections.Mapping):
    def __init__(self):
...
        if islinux or isosx:
            plugins.append('libusb')
            plugins.append('libmtp')

:'-(

I'll fix my upstream patch, too.
Comment 18 Conrad Meyer freebsd_committer 2017-12-26 21:08:51 UTC
Here's my latest upstream patch:
https://github.com/cemeyer/calibre/commit/712edec636fc95e870f7df6ce9d4c2721d6bd816

When I manually patch port files, it rebuilds ok.  Note for PLIST purposes that libusb has been renamed back to the upstream name (libusb) rather than libusb-bsd.

Testing, I no longer see that "cannot load libmtp plugin" exception stack.  However, I also cannot see or send books to my device.  So more work is needed.
Comment 19 Guido Falsi freebsd_committer 2017-12-26 21:34:37 UTC
(In reply to Conrad Meyer from comment #15)

This is strange but seems a QT or X11 problem.

I have calibre running fine on my machine, but I use it only on machines with a full desktop installation.
Comment 20 Conrad Meyer freebsd_committer 2017-12-26 21:43:57 UTC
(In reply to Guido Falsi from comment #19)
> This is strange but seems a QT or X11 problem.

Yes, I agree.
Comment 21 Guido Falsi freebsd_committer 2017-12-26 21:46:33 UTC
(In reply to Conrad Meyer from comment #18)
> Here's my latest upstream patch:
> https://github.com/cemeyer/calibre/commit/
> 712edec636fc95e870f7df6ce9d4c2721d6bd816
> 
> When I manually patch port files, it rebuilds ok.  Note for PLIST purposes
> that libusb has been renamed back to the upstream name (libusb) rather than
> libusb-bsd.

I agree, also because I don't think calibre will find it with a different name.

> 
> Testing, I no longer see that "cannot load libmtp plugin" exception stack. 
> However, I also cannot see or send books to my device.  So more work is
> needed.

You should try running "calibre-debug -g" which launches the calibre gui with debugging on. output goes to stdout/stderr. That should give you some information about what is happening.


You can also try going into the calibre preferences, in the lower part you have a "Miscellaneous" button, and there a "Debug device detection" option, which could be useful in your case.


If you have not done that, you should also check in the plugin list (reachable from preferences) and check the required plugins for your device are all loaded. Some plugins also have some configuration, which you could need to check out.

Hope this helps.
Comment 22 Conrad Meyer freebsd_committer 2017-12-26 21:57:43 UTC
(In reply to Guido Falsi from comment #21)
> You should try running "calibre-debug -g" which launches the calibre gui with
> debugging on. output goes to stdout/stderr. That should give you some
> information about what is happening.

Thanks!  However, nothing interesting is output.  "Started up in 1.04 seconds with 13 books" and then nothing when I plug/remove the device.

> there a "Debug device detection" option, which could be useful in your case.

Sure, I'll try that.  I see:

> Looking for Devices of type MTP_DEVICE
> No MTP devices connected to system
>
> ...
>
> Looking for devices...
> Devices possibly connected: None

Re-running as root, still nothing.
Comment 23 Guido Falsi freebsd_committer 2017-12-26 22:21:09 UTC
(In reply to Conrad Meyer from comment #22)
> (In reply to Guido Falsi from comment #21)
> > You should try running "calibre-debug -g" which launches the calibre gui with
> > debugging on. output goes to stdout/stderr. That should give you some
> > information about what is happening.
> 
> Thanks!  However, nothing interesting is output.  "Started up in 1.04
> seconds with 13 books" and then nothing when I plug/remove the device.
> 
> > there a "Debug device detection" option, which could be useful in your case.
> 
> Sure, I'll try that.  I see:
> 
> > Looking for Devices of type MTP_DEVICE
> > No MTP devices connected to system
> >
> > ...
> >
> > Looking for devices...
> > Devices possibly connected: None
> 
> Re-running as root, still nothing.

So calibre is not detecting your device.

This makes things a little more difficult.

What device are you using with calibre?

Other ideas I get are, maybe some kernel driver is grabbing the device preventing calibre from taking hold of it?

Could you send me info about the device using usbconfig(8) just to get an idea?

Also, while I don't have MPT ereader devices, I used to have an MPT mp3 player. It did require a configuration to be enabled to expose the MPT functionality. I could configured in other operating modes(can't remember the details, it was a few years ago). Maybe your device has similar quirks?
Comment 24 Conrad Meyer freebsd_committer 2017-12-27 01:20:34 UTC
(In reply to Guido Falsi from comment #23)
> What device are you using with calibre?

Comment #3 -- Nook Glowlight Plus, which is basically Android.  It uses MTP.  (FWIW, Calibre on Linux detects the device and works great.)

> Other ideas I get are, maybe some kernel driver is grabbing the device
> preventing calibre from taking hold of it?

Nothing, as far as I can tell.

ugen1.4: <BarnesAndNoble BNRV510> at usbus1

It doesn't show up in devinfo.

> Could you send me info about the device using usbconfig(8) just to get an idea?

$ usbconfig -d ugen1.4 dump_device_desc
ugen1.4: <BarnesAndNoble BNRV510> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (2mA)

  bLength = 0x0012
  bDescriptorType = 0x0001
  bcdUSB = 0x0200
  bDeviceClass = 0x0000  <Probed by interface class>
  bDeviceSubClass = 0x0000
  bDeviceProtocol = 0x0000
  bMaxPacketSize0 = 0x0040
  idVendor = 0x2080
  idProduct = 0x000a
  bcdDevice = 0x9999
  iManufacturer = 0x0002  <BarnesAndNoble>
  iProduct = 0x0003  <BNRV510>
  iSerialNumber = 0x0004  <5072310043863158>
  bNumConfigurations = 0x0001

Anything else from usbconfig?

> Also, while I don't have MPT ereader devices, I used to have an MPT mp3 player.
> It did require a configuration to be enabled to expose the MPT functionality.
> I could configured in other operating modes(can't remember the details, it was
> a few years ago). Maybe your device has similar quirks?

It works from Linux without changing and device configuration.  I don't think there are any options related to MTP to configure on the device.
Comment 25 Guido Falsi freebsd_committer 2017-12-27 14:51:41 UTC
(In reply to Conrad Meyer from comment #24)
> (In reply to Guido Falsi from comment #23)
> > What device are you using with calibre?
> 
> Comment #3 -- Nook Glowlight Plus, which is basically Android.  It uses MTP.
> (FWIW, Calibre on Linux detects the device and works great.)
> 

I don't doubt it works on linux or windows, I was asking to try to see if it did ring some bell.


> 
> > Could you send me info about the device using usbconfig(8) just to get an idea?
> 
> $ usbconfig -d ugen1.4 dump_device_desc
> ugen1.4: <BarnesAndNoble BNRV510> at usbus1, cfg=0 md=HOST spd=HIGH
> (480Mbps) pwr=ON (2mA)
> 
>   bLength = 0x0012
>   bDescriptorType = 0x0001
>   bcdUSB = 0x0200
>   bDeviceClass = 0x0000  <Probed by interface class>
>   bDeviceSubClass = 0x0000
>   bDeviceProtocol = 0x0000
>   bMaxPacketSize0 = 0x0040
>   idVendor = 0x2080
>   idProduct = 0x000a
>   bcdDevice = 0x9999
>   iManufacturer = 0x0002  <BarnesAndNoble>
>   iProduct = 0x0003  <BNRV510>
>   iSerialNumber = 0x0004  <5072310043863158>
>   bNumConfigurations = 0x0001
> 
> Anything else from usbconfig?
> 
> > Also, while I don't have MPT ereader devices, I used to have an MPT mp3 player.
> > It did require a configuration to be enabled to expose the MPT functionality.
> > I could configured in other operating modes(can't remember the details, it was
> > a few years ago). Maybe your device has similar quirks?
> 
> It works from Linux without changing and device configuration.  I don't
> think there are any options related to MTP to configure on the device.

Unluckily I'm getting out of ideas. I'll investigate the calibre detection code.

I'll followup once I have something new.

We are clearly missing something.

One thing that is needed for sure is write access to the proper device, which can be warranted using devfs or devd rules. But you also tried as root which should avoid such an issue.
Comment 26 Conrad Meyer freebsd_committer 2017-12-27 14:55:38 UTC
(In reply to Guido Falsi from comment #25)
> Unluckily I'm getting out of ideas. I'll investigate the calibre detection code.

Yep, me too :-(

> One thing that is needed for sure is write access to the proper device, which
> can be warranted using devfs or devd rules. But you also tried as root which
> should avoid such an issue.

Yeah, I also tried chmod'ing the USB device go+rw.  No luck.  I will try some of the programs associated with the libmtp port and see if they can see the device.
Comment 27 Conrad Meyer freebsd_committer 2017-12-27 14:58:23 UTC
mtp-detect as root can detect the device.  Not as non-root (without chmodded devfs entry).  With chmod go+rw, non-root can detect with mtp-detect.

mtp-folders and mtp-files can enumerate content just fine.  Hm.
Comment 28 Guido Falsi freebsd_committer 2017-12-27 15:04:32 UTC
(In reply to Conrad Meyer from comment #27)
> mtp-detect as root can detect the device.  Not as non-root (without chmodded
> devfs entry).  With chmod go+rw, non-root can detect with mtp-detect.
> 
> mtp-folders and mtp-files can enumerate content just fine.  Hm.

This indicates the problem is definitely in the calibre device enumeration code.

I alreaady worked on that, but it was some time ago. I need to take a good look.
Comment 29 Conrad Meyer freebsd_committer 2017-12-27 15:04:46 UTC
Ahaha.  Maybe this is the problem?

https://github.com/kovidgoyal/calibre/blob/master/src/calibre/devices/mtp/unix/devices.c
Comment 30 Conrad Meyer freebsd_committer 2017-12-27 15:05:06 UTC
Hm, that doesn't explain why it works on Linux, though.
Comment 31 Conrad Meyer freebsd_committer 2017-12-27 15:07:16 UTC
Also, the device is listed in the included header:
https://github.com/kovidgoyal/calibre/blob/master/src/calibre/devices/mtp/unix/upstream/music-players.h#L2917
Comment 32 Conrad Meyer freebsd_committer 2017-12-27 15:09:52 UTC
Oh, this might be it:

https://github.com/kovidgoyal/calibre/blob/master/src/calibre/devices/mtp/unix/driver.py#L53

The sysfs enumeration is probably Linux only.  We don't populate sysfs entries for USB devices, do we?  https://github.com/kovidgoyal/calibre/blob/master/src/calibre/devices/mtp/unix/sysfs.py
Comment 33 Conrad Meyer freebsd_committer 2017-12-27 15:13:49 UTC
It seems totally silly to use sysfs for detection when libmtp can enumerate devices itself:

https://github.com/mmalecki/libmtp/blob/master/examples/detect.c (LIBMTP_Detect_Raw_Devices())

Heck, the calibre libmtp python wrapper even invokes that API already.  It just doesn't export it for enumeration purposes:  https://github.com/kovidgoyal/calibre/blob/master/src/calibre/devices/mtp/unix/libmtp.c#L209

We should probably submit a patch upstream to detect MTP devices using libmtp, cross-platform.  There's no need for separate OS X and Linux (and FreeBSD) enumeration.
Comment 34 Conrad Meyer freebsd_committer 2017-12-27 15:31:23 UTC
(May also be that this is not sufficient: https://github.com/kovidgoyal/calibre/blob/master/src/calibre/devices/scanner.py#L169  .  Just guessing, no time to work on it today.)
Comment 35 Guido Falsi freebsd_committer 2017-12-27 15:46:25 UTC
(In reply to Conrad Meyer from comment #33)
> It seems totally silly to use sysfs for detection when libmtp can enumerate
> devices itself:
> 
> https://github.com/mmalecki/libmtp/blob/master/examples/detect.c
> (LIBMTP_Detect_Raw_Devices())
> 
> Heck, the calibre libmtp python wrapper even invokes that API already.  It
> just doesn't export it for enumeration purposes: 
> https://github.com/kovidgoyal/calibre/blob/master/src/calibre/devices/mtp/
> unix/libmtp.c#L209
> 
> We should probably submit a patch upstream to detect MTP devices using
> libmtp, cross-platform.  There's no need for separate OS X and Linux (and
> FreeBSD) enumeration.

I like your idea. Problem is calibre detection routine is a little convoluted. I need time to study it to see how I can help you in this.
Comment 36 Guido Falsi freebsd_committer 2017-12-27 21:52:52 UTC
Judging from the comments in the code, the author is using sysfs to avoid probing the devices, since, I imagine, the linux kernel has already probed them to populate sysfs.

Also the FreeBSDScanner is looking only for umass devices, so it ignores anything else.

As I said the code looks a little convoluted, but the easiest way seems to be to replicate the same steps in FreeBSD.

So the plan is, first to teach FreeBSDScanner to get all USB devices, like the linux one does.

After that we need a way to check for MTP capable devices. My idea is that the easiest way is to add such capability to the libmtp wrapper, and import that method in the python code and use it there. Are you proficient in coding python libraries in C?

Once that is done everything should work, at least in theory and save for errors.

I must find some time to start hacking around at the FreeBSDScanner function.
Comment 37 Conrad Meyer freebsd_committer 2017-12-28 05:23:08 UTC
(In reply to Guido Falsi from comment #36)
> Judging from the comments in the code, the author is using sysfs to avoid
> probing the devices, since, I imagine, the linux kernel has already probed
> them to populate sysfs.

Sure, but, how many USB devices are in a typical system anyway?  For me it is often 1-3.  Reprobing seems like it would have essentially no cost.

> So the plan is, first to teach FreeBSDScanner to get all USB devices, like the
> linux one does.

Sounds good to me!

> After that we need a way to check for MTP capable devices. My idea is that the
> easiest way is to add such capability to the libmtp wrapper, and import that
> method in the python code and use it there. Are you proficient in coding python
> libraries in C?

Yes, that seems like a good method to me.  Yep, I can do the python C library part.
Comment 38 Guido Falsi freebsd_committer 2018-01-05 14:02:15 UTC
Created attachment 189442 [details]
Update to version including upstreamed changes

Upstream calibre has released a new version including the changes (thanks you Conrad for taking care of upstreaming them).

This is the patch I'll going to commit to update the port.

I've added a note in pkg-message with steps needed to have access to USB MPT devices.

Could you please test it before I commit, so we make sure it all works fine and I can add an announce to the commit message about the new MPT support?

Thanks in advance.
Comment 39 Guido Falsi freebsd_committer 2018-01-05 14:05:01 UTC
The patch contains an error, MPT in place of MTP, I've fixed that locally.
Comment 40 Conrad Meyer freebsd_committer 2018-01-05 18:41:06 UTC
This part:


+Then, add the following to /etc/devfs.rules, or create this file
+if it does not exist yet:
+
+[system=10] add path 'usb/*' mode 0660 group operator
+
+To load these new rules, add the following to /etc/rc.conf:
+
+devfs_system_ruleset="system"
+
+Then, restart devfs
+
+# service devfs restart


Doesn't seem to work:

$ ls -l /dev/usb/*
crw-------  1 root  operator  0x8b Jan  5 10:33 /dev/usb/1.4.0
crw-------  1 root  operator  0x8d Jan  5 10:33 /dev/usb/1.4.1
crw-------  1 root  operator  0x8e Jan  5 10:33 /dev/usb/1.4.2


Here's what attach looks like:

$ cat /var/run/devd.pipe
...
!system=DEVFS subsystem=CDEV type=CREATE cdev=usb/1.4.0
!system=DEVFS subsystem=CDEV type=CREATE cdev=ugen1.4
!system=DEVFS subsystem=CDEV type=CREATE cdev=usb/1.4.1
!system=DEVFS subsystem=CDEV type=CREATE cdev=usb/1.4.2
!system=USB subsystem=DEVICE type=ATTACH ugen=ugen1.4 cdev=ugen1.4 vendor=0x2080 product=0x000a devclass=0x00 devsubclass=0x00 sernum="5072310043863158" release=0x9999 mode=host port=2 parent=ugen1.2
!system=USB subsystem=INTERFACE type=ATTACH ugen=ugen1.4 cdev=ugen1.4 vendor=0x2080 product=0x000a devclass=0x00 devsubclass=0x00 sernum="5072310043863158" release=0x9999 mode=host interface=0 endpoints=3 intclass=0xff intsubclass=0xff intprotocol=0x00
? at bus=1 hubaddr=2 port=2 devaddr=4 interface=0 ugen=ugen1.4 vendor=0x2080 product=0x000a devclass=0x00 devsubclass=0x00 devproto=0x00 sernum="5072310043863158" release=0x9999 mode=host intclass=0xff intsubclass=0xff intprotocol=0x00 on uhub4


$ devfs rule show system
devfs rule: error converting to integer: system
$ devfs rule showsets
1
2
3
4
10
$ devfs rule show 10
devfs rule: ioctl DEVFSIO_RGETNEXT: No such file or directory


Anyway, based on what /etc/defaults/devfs.rules looks like, I'm guessing a newline is missing between the [system=10] part and the "add path" part.  And indeed, that change fixes it.
Comment 41 Conrad Meyer freebsd_committer 2018-01-05 18:43:27 UTC
By the way, do you know anything about this?

[1/1] Deleting files for calibre-3.14.0_1:  99%
pkg: /usr/local/share/mime/application/epub+zip.xml different from original checksum, not removing
[1/1] Deleting files for calibre-3.14.0_1:  99%
pkg: /usr/local/share/mime/application/x-mobipocket-ebook.xml different from original checksum, not removing
Comment 42 Conrad Meyer freebsd_committer 2018-01-05 19:04:45 UTC
(In reply to Guido Falsi from comment #38)
Patch applies cleanly and builds fine.  With (corrected) devfs.rules and logout/login to apply operators group change, 'calibre-debug -d' successfully detects the device as my ordinary user and 'calibre-debug -g' launches successfully and can manage the device.  All working!
Comment 43 Guido Falsi freebsd_committer 2018-01-05 19:13:42 UTC
(In reply to Conrad Meyer from comment #40)

> Anyway, based on what /etc/defaults/devfs.rules looks like, I'm guessing a
> newline is missing between the [system=10] part and the "add path" part. 
> And indeed, that change fixes it.

Thanks for spotting that. Must have happened while copying text!
Comment 44 Guido Falsi freebsd_committer 2018-01-05 19:14:27 UTC
(In reply to Conrad Meyer from comment #41)
> By the way, do you know anything about this?
> 
> [1/1] Deleting files for calibre-3.14.0_1:  99%
> pkg: /usr/local/share/mime/application/epub+zip.xml different from original
> checksum, not removing
> [1/1] Deleting files for calibre-3.14.0_1:  99%
> pkg: /usr/local/share/mime/application/x-mobipocket-ebook.xml different from
> original checksum, not removing

Not much. I always forget to check. I'll do that now, since I have time.
Comment 45 Guido Falsi freebsd_committer 2018-01-05 19:40:12 UTC
(In reply to Guido Falsi from comment #44)
> (In reply to Conrad Meyer from comment #41)
> > By the way, do you know anything about this?
> > 
> > [1/1] Deleting files for calibre-3.14.0_1:  99%
> > pkg: /usr/local/share/mime/application/epub+zip.xml different from original
> > checksum, not removing
> > [1/1] Deleting files for calibre-3.14.0_1:  99%
> > pkg: /usr/local/share/mime/application/x-mobipocket-ebook.xml different from
> > original checksum, not removing
> 
> Not much. I always forget to check. I'll do that now, since I have time.

After a few tests I can say it's modified by update-mime-database from misc/shared-mime-info. Actually it's mangled by it for reasons which are not clear to me.

It is added as a post install command by pkg create, following it's own logic. I think it adds that for any port adding mime info automatically.

I'm trying to understand why update-mime-database modifies these files, it should actually just read them.
Comment 46 Guido Falsi freebsd_committer 2018-01-05 20:18:21 UTC
(In reply to Guido Falsi from comment #45)
> (In reply to Guido Falsi from comment #44)
> > (In reply to Conrad Meyer from comment #41)
> > > By the way, do you know anything about this?
> > > 
> > > [1/1] Deleting files for calibre-3.14.0_1:  99%
> > > pkg: /usr/local/share/mime/application/epub+zip.xml different from original
> > > checksum, not removing
> > > [1/1] Deleting files for calibre-3.14.0_1:  99%
> > > pkg: /usr/local/share/mime/application/x-mobipocket-ebook.xml different from
> > > original checksum, not removing
> > 
> > Not much. I always forget to check. I'll do that now, since I have time.
> 
> After a few tests I can say it's modified by update-mime-database from
> misc/shared-mime-info. Actually it's mangled by it for reasons which are not
> clear to me.
> 
> It is added as a post install command by pkg create, following it's own
> logic. I think it adds that for any port adding mime info automatically.
> 
> I'm trying to understand why update-mime-database modifies these files, it
> should actually just read them.

I think I found the cause.

The port is actually installing some files which are automatically generated based on another "master" file. The master file is correctly installed, the generated one should not end up in the package.

The difference in the content of the files is caused by update-mime-database giving precedence to another global configuration for those two formats.

Solution is avoid installing the autogenerated files.
Comment 47 commit-hook freebsd_committer 2018-01-06 09:27:03 UTC
A commit references this bug:

Author: madpilot
Date: Sat Jan  6 09:26:25 UTC 2018
New revision: 458237
URL: https://svnweb.freebsd.org/changeset/ports/458237

Log:
  - Update calibre to 3.15.0
  - Avoid packaging mime database auto generated files

  This version enables MTP devices support on FreeBSD.

  Many thanks to cem@ for reporting the lack of support, doing most
  of the coding, taking care of upstreaming patches and testing. Also
  thanks for reminding me to fix the mime database problem!

  PR:		224580
  Submitted by:	cem@

Changes:
  head/deskutils/calibre/Makefile
  head/deskutils/calibre/distinfo
  head/deskutils/calibre/files/patch-setup_build__environment.py
  head/deskutils/calibre/files/patch-src_calibre_linux.py
  head/deskutils/calibre/pkg-message
  head/deskutils/calibre/pkg-plist
Comment 48 Guido Falsi freebsd_committer 2018-01-06 09:28:00 UTC
Update and fixes committed.

Thanks a lot for reporting, helping and testing!
Comment 49 Conrad Meyer freebsd_committer 2018-01-06 16:29:33 UTC
Awesome!  I am happy to see it fixed in ports :-).