Bug 246121 - [bhyve][PATCH] Append Keyboard Layout specified option for using VNC.
Summary: [bhyve][PATCH] Append Keyboard Layout specified option for using VNC.
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: Unspecified
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-virtualization mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-05-02 12:09 UTC by Koine Yuusuke
Modified: 2020-06-05 03:15 UTC (History)
6 users (show)

See Also:


Attachments
[bhyve][PATCH] Append Keyboard Layout specified option for using VNC. (9.45 KB, text/plain)
2020-05-02 12:09 UTC, Koine Yuusuke
no flags Details
[bhyve][PATCH] Append Keyboard Layout specified option for using VNC. (21.34 KB, text/plain)
2020-05-21 13:40 UTC, Koine Yuusuke
no flags Details
[bhyve][PATCH] Support QEMU Extended Key Event Message and Append keyboard layout specified option. (25.63 KB, text/plain)
2020-05-29 10:47 UTC, Koine Yuusuke
no flags Details
[PATCH] supporting QEMU Extended Keyboard Event Message (5.26 KB, text/plain)
2020-06-03 13:24 UTC, Koine Yuusuke
no flags Details
[PATCH] Append bhyve -k option for specified keyboard layout with layout setting files every languages. (22.67 KB, text/plain)
2020-06-03 13:28 UTC, Koine Yuusuke
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Koine Yuusuke 2020-05-02 12:09:04 UTC
Created attachment 214027 [details]
[bhyve][PATCH] Append Keyboard Layout specified option for using VNC.

When a guest OS is loaded using UEFI and connected with VNC, if the keyboard on the VNC client side is other than the US keyboard, some keys input may not be p
erformed correctly.
For example, if you press the '@' key on a Japanese keyboard, the number '2' will be entered on the guest OS.
The cause is that the '@' key on the US keyboard is assigned to Shift + '2' keys.
In addition, there is a problem that keys that do not exist on the US keyboard cannot be entered.
  (example, Zenkaku-Hankaku key in the Japanese Keyboard)

In the current bhyve(13-Current & 12.1-RELEASE-p3), in the virtual PS2 keyboard driver, there is a conversion table that converts from the key entered from the
 VNC client side to the virtual keyboard ScanCode for the Guest OS.
However, since this conversion table is set for US keyboards, the above problem will occur if the client side is not a US keyboard.

Therefore, I created a patch that allows the conversion table to be set in the external configuration file for each keyboard layout, and that the configuration
 file (=keyboard layout) can be specified by the option '-k' of the bhyve command.

The following shows how to apply the patch and how to use it.
Please merge it into the source tree if possible.

I'm new to creating a patch for the FreeBSD source tree, and I'm not sure if it's correct to submit to Bugzilla.
If it's wrong to submit it to Bugzilla, it would be very helpful if you could tell me how to submit it.


A. How to apply the patch ----------------------------------
  1. cd /usr/src/usr.sbin
  2. patch -u < bhyve_kbdlayout_fbsd13c.patch
  3. sh bhyve_kbdlayout.shar
  4. cd /usr/src/usr.sbin/bhyve
  5. make
  6. make install

  * If you use the FreeBSD 12.1-RELEASE, please replace the "fbsd13c" of the above No.2 to the "fbsd121r".


B. How to specified the keyboard layout --------------------
  If you use the bhyve command directly, specify the '-k layout' option for the bhyve command.
  You can specify the "layout" in the file name stored in /usr/share/bhyve/kbdlayout dir.
  If no '-k' option is specified, the US keyboard (default) is assumed to be selected.

  Others, If you use the bhyve via the vm-bhyve package, specify the following lines for the Guest OS configure file.
     bhyve_options="-k layout"

  
C. Request for Coooperation (The remaining tasks) ----------
  Currently, the keyboard layout setting file can only be created for Japanese keyboards.
   (Because, I have only the Japanese Keyboard...)

  If you would like to adopt this patch, I would like to ask those who use keyboards of the corresponding languages to create keyboard layout setting files oth
er than English and Japanese keyboards.
  See the 'default' file in the /usr/share/bhyve/kbdlayout dir. for how to create it.
Comment 1 Aleksandr Fedorov 2020-05-04 12:58:50 UTC
It looks interesting, but can we use existing layout files: /usr/share/vt/keymaps or /usr/share/syscons/keymaps?

See format description: man 5 kbdmap
Comment 2 Koine Yuusuke 2020-05-05 11:51:05 UTC
Thank you for you interest in the patch.

Unfortunately, I understand that the your suggestions are difficult for the following reasons.

The virtual PS2 keyboard driver of the bhyve has a table for converting ASCII code for keys that can be expressed in ASCII characters and Xorg keysym code for keys that cannot be expressed in ASCII characters (example Enter or Escape key,etc) to ScanCode Set2 of the PS2 keyboard.
This patch updates this conversion table based on the layout setting file.

The /usr/share/vt/keymaps/* file contain the correspondence of ASCII characters or key names (key names is unique to vt/syscons, not Xorg keysym) corresponding to the scancode (for vt/syscons) converted from ScanCode Set2 of the PS2 keyboard device.
In addition, I confirmed that the conversion algorithm from ScanCode Set2 of the PS2 keyboard to scancode (for vt/syscons) is implemented in /usr/src/sys/dev/atkbdc/atkbd.c, so I think that it is not impossible to use the /usr/share/vt/keymaps/* file as the basis for the keys that can be represented by ASCII characters.
(Since it has not been examined in detail, I don't understand whether ScanCode Set2 of the PS2 keyboard can actually be converted from scancode (for vt/syscons) to the reverse.)

On the other hand, a key that cannot be represented by ASCII characters cannot be supported because the value of Xorg keysym value doesn't exist in /usr/share/vt/keymaps/* file.
 (For reference, the value of Xorg keysym value is in the /usr/local/include/X11/keysymdef.h)

Therefore, even if the key layout conversion information that can be expressed in ASCII characters is acquired from /usr/share/vt/keymaps/* file, the key that cannot be expressed in ASCII characters cannot be acquired from /usr/share/vt/keymaps/* file, and thus I think that a layout configuration file for each language is required.

If you have a good idea to use /usr/share/vt/keymaps/* file, please let me know.
 (I also don't want to have a layout configuration file for bhyve apart from /usr/share/vt/keymaps/* file.
  But I did this because I couldn't think of any other simple way...)

 - Koine Yuusuke (koinec)
Comment 3 Peter Grehan freebsd_committer 2020-05-13 10:55:22 UTC
Another way to solve this is to implement the "QEMU Extended Key Event Message" in the server (https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#qemu-extended-key-event-message)

This allows scancodes to be sent end-end so avoids information loss and locale mis-translation when converting VNC keycodes to PS2 scancodes.

The downside is that client support is patchy :( It's in recent versions of TigerVNC and originated fron NoVMC, but isn't in VNCViewer, tightVNC or UltraVNC :(

However, it avoids the problem of having to create per-language maps.
Comment 4 Lars Engels freebsd_committer 2020-05-13 13:20:11 UTC
There's at least a workaround to make non-english keyboards work:

In the bhyve client configure the keyboard to your locale, e.g. de, jp, ...

On the host running the VNC client change your keyboard setting to "us" e.g. "setxkbmap us".

Then connect using TigerVNC or probably other clients. Inside the VNC client all keys e.g. German umlauts should work normally.
Comment 5 Koine Yuusuke 2020-05-21 13:40:44 UTC
Created attachment 214725 [details]
[bhyve][PATCH] Append Keyboard Layout specified option for using VNC.

Thank you for the various ideas!

As a result of thinking, I think that the method that has the keyboard-layout setting file is better.
However, I tried to automatically generate the keyboard-layout setting file for each language from the file existing /usr/share/vt/kbdmap for vt, so update the generated keyboard-layout patch files for each language.
Now that I can enter the following keys on the keyboards of all languages supported by vt, what about?

A. How to apply the patch ----------------------------------
  1. cd /usr/src/usr.sbin
  2. patch -u < bhyve_kbdlayout_fbsd13c.patch
  3. sh bhyve_kbdlayout.shar           (*Updated the bhyve_kbdlayou.shar file in this time.)
  4. cd /usr/src/usr.sbin/bhyve
  5. make
  6. make install

  * If you use the FreeBSD 12.1-RELEASE, please replace the "fbsd13c" of the above No.2 to the "fbsd121r".

B. Supoorting Keyboard map ---------------------------------
  You can use the following keyboard language layout.
   (Indicates the string that can be specified in the bhyve -k option.)
      am / be / be_acc / bg_bds / bg_phonetic / br / br_noacc / by
      ca / ca-fr / centraleuropean / centraleuropean_qwerty
      ch / ch_acc / ch-fr / ch-fr_acc / ch_macbook_acc / colemak_acc
      cz / de / de_acc / de_noacc / dk / dk_acc / dk_macbook
      ee / es / es_acc / es_dvorak / fi
      fr / fr_acc / fr_dvorak / fr_dvorak_acc / fr_macbook
      gr / gr_101_acc / gr_elot_acc / hr / hu_101 / hu_102
      il / is / is_acc / it / kz_io / kz_kst / lt
      latinamerican / latinamerican_acc / nl / no / no_dvorak / nordic_asus-eee
      pl / pl_dvorak / pt / pt_acc / ru / ru_shift / ru_win / se / si / sk
      tr / tr_f / ua / ua_shift_alt / uk / uk_capsctrl / uk_dvorak / uk_macbook
      us / us_acc / us_ctrl / us_emacs / us_macbook / us_unix
      us_dvorak / us_dvorakl / us_dvorakp / us_dvorakr / us_dvorakx
      jp / jp_capsctrl

C. Supporting Key with the above Keyboard-Layout file. ----
  Alphabetical Key ( [a-zA-Z] )
  Numeric Key      ( [0-9] )
  Sign mark Key    ( !"#$%&'()+-*/=^~\|@`[]{};:,.<>?_ )
  Enter / Escape / Tab / Shift(L/R) / Ctrl.(L/R) / Alt(L/R) / Home / End / Ins. / BackSpace / Delete
  Up Arrow / Down Arrow / Left Arrow / Right Arrow / PgUp / PgDown
  F1-12 / WinKey(L/R)

    * jp / jp_capsctl supports Japanese keyboard specific keys.

D. Thoghts on the ideas you taught ------------------------
 1. The workaround by executing "setxkbmap us" on the VNC client side.
   This method allows the guest OS to enter the correct key input, but there is a problem that the key cannot be input correctly on the client side(HyberVisor) terminal.
   Therefore, it is not possible to use the FreeBSD terminal on which bhyve runs and the guest OS as the same time.
   I knew it was also introduced as a workaround on other sites, but I patched this thread because of the above issue.

 2. The use of QEMU Extended Key Event Message
   I understand that this method is a drastic measure to avoid using the keyboard-layout files, and I would like to make the patch with it if possible ...
   However, in the current VNC clients that exist the FreeBSD ports, tigerVNC doesn't support this message because it is an old version, and it seems that tightVNC & ssvnc doesn't support this message in the first place.
   If the VNC clients that exist the FreeBSd ports supprots this message, I would like to try to support this message, but I can not support it because there is no VNCclient currently. And as far as I can see the bhyve source tree, I think the difficulty level of the modification is high.

 3. This proposal
   When we aim to make it easy to use any language using the current VNCclient on FreeBSD ports, I think that it is unavoidable to use the keyboard-layout setting file.
   However, it is difficult to create the keyboard-layout files for each launguage from the beginning, so the minimum keys such as alphabets, numbers, and symbols were automatically generated.
   I would like to ask if this method is acceptable, test the generating keyboard-layout setting file for each language, and add a unique key definition for each language.
Comment 6 Peter Grehan freebsd_committer 2020-05-28 11:44:47 UTC
TigerVNC in ports is v1.10.1, which is the latest official release. The Qemu key event code has been in there for a couple of years now.

I'll have a look at how much work it is to support that extension, but I grant that a) it isn't supported by other VNC clients, and b) the performance of TigerVNC is worse than TightVNC. So, the keymap change is still likely worthwhile. I'll take a closer look at your patch.
Comment 7 Koine Yuusuke 2020-05-29 10:47:36 UTC
Created attachment 215004 [details]
[bhyve][PATCH] Support QEMU Extended Key Event Message and Append keyboard layout specified option.

Thank you for telling me and I'm sorry for my lack of research.

With you help, I found that the TigerVNC supports QEMU Extended Key Event Message, so I immediately tried to create a patch.
As a result, when using TigerVNC, it is possible to support keys unique of each language without depending the keyboard layout setting file or specifying bhyve -k option.
However, I would like to preserve the keyboard layout configuration file and the bhyve -k option feature to allow use by VNC clients that don't support QEMU Extended Key Event Messages (such as tightVNC).

By the last time, the keyboard layout setting file should have been able to input alphabets, numbers, symbols, and a part of functional key(e.g. Enter/Space) in every languages keyboard, so I think that support has advanced considerably, but what about?


A. How to apply the NEW patch --------------------------------------------
  Same as last time. (See above)

B. Usage -----------------------------------------------------------------
  * VNC clients with supported the QEMU Extended Key Event Mesage (TigerVNC)
      Just use it as before.
      The keyboard layout of each language is automatically supported.
      Even if the -k option is specified, it will be ignored.

  * VNC clients WITHOUT supported the message (tightVNC, ssvnc, etc)
      You need to specify the keyboard layout name by the -k option.
      The method of specify the -k option is same as last time.
      If not specified, the US keyboard layout is assumed to have been specified.

I apologize for fixing the path many times, but I appreciate your cooperation.
Comment 8 Peter Grehan freebsd_committer 2020-06-01 12:30:34 UTC
Thanks - great work :)

I'll split this out into 2 parts. The first will be support for the qemu extended key event message, since this requires no documentation, and can provide a solution today for those willing to use TigerVNC (and it will also provide some impetus to fix the perf issues with TigerVNC). I'll put up a review for this shortly.

The second will be adding support for the translation-table file.
Comment 9 Koine Yuusuke 2020-06-03 13:24:46 UTC
Created attachment 215195 [details]
[PATCH] supporting QEMU Extended Keyboard Event Message

Thank you for everything.

The patch has been divided into two parts, so please use it if you like.

The first patch is "bhyve_qemu_ext_key_event_msg.tgz" for supporting QEMU Extended Keyboard Event Message.
It can be applied by executing the following command on /usr/src/usr.sbin
   patch -u < bhyve_qemu_ext_key_event_msg_fbsd13c.patch

   * If you use the FreeBSD 12.1-RELEASE, please replace the "fbsd13c" of the above to the "fbsd121r".
Comment 10 Koine Yuusuke 2020-06-03 13:28:10 UTC
Created attachment 215196 [details]
[PATCH] Append bhyve -k option for specified keyboard layout with layout setting files every languages.

Then, the second patch is "bhyve_kbdlayout_option.tgz" for supporting bhyve -k option that specify the keyboard layout name.
This patch can be applied be executing the following command.
   cd /usr/src/usr.sbin
   patch -u < bhyve_kbdlayout_option_fbsd13c.patch
   sh bhyve_kbdlayout_layoutfile.shar
   cd /usr/src/share
   patch -u < bhyve_kbdlayout_vmrunsh_fbsd13c.patch

In addition, the second patch has the following changes from the previous patch.
   * The first patch (for supporting QEMU Extended Keyboard Event Message) must be applied before applying this patch.
   * Fix an issue where the latest bhyve source tree (only FreeBSD 13.0-current) could not be patched correctly. (SORRY!)
   * Modified so that -k option can be specified from /usr/share/examples/bhyve/vmrun.sh.
      (However, vmrun.sh is only modified.)

If there is something, please point out.
Comment 11 Peter Grehan freebsd_committer 2020-06-05 03:15:33 UTC
(In reply to Koine Yuusuke from comment #9)

Thanks for the patch. I'd already broken out that section from your previous one so the review might look slightly different.

One issue I had with the qemu ext keyevent is that TigerVNC generates scanset 1 codes (with freebsd/macos/windows clients), whereas the atkbd emulation is expecting scanset 2 codes in the FIFO. There's a simple 1:1 translation between scanset 1 and 2, so I'll add a table for that conversion.