Bug 221676

Summary: libxo fails to format certain fixed values correctly (eg: arp: invalid json output from libxo when bridge is present)
Product: Base System Reporter: Jimmy Olgeni <olgeni>
Component: binAssignee: Phil Shafer <phil>
Status: Closed FIXED    
Severity: Affects Some People CC: duncan, garga, koobs, marcel, net, phil, re, rgrimes, wes
Priority: --- Keywords: easy
Version: CURRENTFlags: koobs: mfc-stable11+
Hardware: Any   
OS: Any   
Attachments:
Description Flags
Adds the quote directive to xo fields to make sure that strings are quoted in json output koobs: maintainer-approval-

Description Jimmy Olgeni freebsd_committer 2017-08-20 19:34:33 UTC
Trying to get json output from arp(8):

    # arp -a --libxo json | jq .
    parse error: Invalid numeric literal at line 1, column 170

Indeed, the output document was this:

{"__version": "1", "arp": {"arp-cache": [{"hostname":"?","ip-address":"172.16.1.1","mac-address":"02:c9:3a:59:26:00","interface":"bridge0","permanent":true,"type":bridge}, {"hostname":"olgeni.home.olgeni.com","ip-address":"192.168.0.1","mac-address":"bc:ae:c5:5b:e5:e5","interface":"re0","permanent":true,"type":"ethernet"}, {"hostname":"?","ip-address":"192.168.0.254","mac-address":"4c:60:de:f9:44:46","interface":"re0","expires":1173,"type":"ethernet"}]}
}

Everything looks ok, but the 'bridge' string is not quoted.
Comment 1 Duncan Paterson 2017-08-26 16:55:33 UTC
Created attachment 185787 [details]
Adds the quote directive to xo fields to make sure that strings are quoted in json output

I can reproduce this in general for libxo. 

The number detecting heuristic it uses allows for C style numeric suffices (for example 34f, 12d). This allows any strings ending the characters in the set [diouDOUeEfFgG] to be treated as numbers (left unquoted in json). This has two drawbacks firstly leaving strings such as bridge unquoted because it ends in e, secondly it allows numbers with C style suffices to be output unqoted these are not valid json.

for example:
{
  "key": 35f
}

is invalid.

I've added the quote directive to all of the relevant xo_emit calls as per the documentation on libxo at http://juniper.github.io/libxo/libxo-manual.html.
Comment 2 Duncan Paterson 2017-08-26 16:56:26 UTC
I've added a patch which should fix this problem for the arp application. I think this may apply to other applications using libxo as well.
Comment 3 Wesley Moore 2018-04-28 07:51:29 UTC
I just hit the same issue. Looking at the current code in head[1] it looks like the patch still needs to be applied. Anything I can do to help make that happen?

[1]: https://svnweb.freebsd.org/base/head/usr.sbin/arp/arp.c?revision=331714&view=markup#l670
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2018-05-15 05:40:36 UTC
CC committer of libxo support (base r316592) for arp, re@ for 11.2-STABLE (and subsequent RELEASE) merge approval
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2018-05-15 05:52:11 UTC
@Jimmy Now that there is a patch, and if you have the cycles, feel free to assign yourself and shepherd the change in with a phabricator review and finding someone to review it.

While I'm here cc Phil and Marcel, who have updated libxo in base in the past, and may be able to help review (along with Renato)
Comment 6 Phil Shafer freebsd_committer 2018-05-15 14:01:09 UTC
No, this is very much a libxo bug.

I'm looking at the "fmt" to see if it's a numeric format, for example, "%32d" or "%f", but I don't handle the case where it's a fixed value correctly, so the "{:type/ethernet}" works but "{:type/bridge}" fails, since "e" is a numeric format type.  I'll add a check to handle this (in xo_format_value()).

Apologies for the issue.

Thanks,
 Phil
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2018-05-16 02:29:07 UTC
Comment on attachment 185787 [details]
Adds the quote directive to xo fields to make sure that strings are quoted in json output

Bug is in libxo, not specific to arp.
Comment 8 Phil Shafer freebsd_committer 2018-05-21 18:08:37 UTC
Have fix in libxo github repo:

https://github.com/Juniper/libxo/commit/27d3021cc3cc8cfbe9ddee5930cd7a9afea8f68f#diff-5a0d468963477f7daedb8308c219dd80

I'll spin a release and import it into HEAD this afternoon.

Thanks,
 Phil
Comment 9 commit-hook freebsd_committer 2018-05-23 01:21:07 UTC
A commit references this bug:

Author: phil
Date: Wed May 23 01:20:32 UTC 2018
New revision: 334068
URL: https://svnweb.freebsd.org/changeset/base/334068

Log:
  Import libxo-0.9.0:
  - Add xo_format_is_numeric() with improved logic to decide if format
    strings are numeric, so json output quotes them
  - Convert docs to sphinx/rst
  - update tests

  Includes fix for PR 221676:
  https://github.com/Juniper/libxo/commit/27d3021cc3cc8cfbe9ddee5930cd7a9afea8f68f#diff-5a0d468963477f7daedb8308c219dd80

  PR:		 221676
  MFC after:	5 days

Changes:
_U  head/contrib/libxo/
  head/contrib/libxo/configure.ac
  head/contrib/libxo/doc/Makefile.am
  head/contrib/libxo/doc/_static/
  head/contrib/libxo/doc/_templates/
  head/contrib/libxo/doc/api.rst
  head/contrib/libxo/doc/conf.py
  head/contrib/libxo/doc/example.rst
  head/contrib/libxo/doc/faq.rst
  head/contrib/libxo/doc/field-formatting.rst
  head/contrib/libxo/doc/field-modifiers.rst
  head/contrib/libxo/doc/field-roles.rst
  head/contrib/libxo/doc/format-strings.rst
  head/contrib/libxo/doc/formatting.rst
  head/contrib/libxo/doc/getting.rst
  head/contrib/libxo/doc/howto.rst
  head/contrib/libxo/doc/index.rst
  head/contrib/libxo/doc/intro.rst
  head/contrib/libxo/doc/libxo-manual.html
  head/contrib/libxo/doc/options.rst
  head/contrib/libxo/doc/xo.rst
  head/contrib/libxo/doc/xohtml.rst
  head/contrib/libxo/doc/xolint.rst
  head/contrib/libxo/doc/xopo.rst
  head/contrib/libxo/libxo/libxo.c
  head/contrib/libxo/tests/core/saved/test_01.E.out
  head/contrib/libxo/tests/core/saved/test_01.H.out
  head/contrib/libxo/tests/core/saved/test_01.HIPx.out
  head/contrib/libxo/tests/core/saved/test_01.HP.out
  head/contrib/libxo/tests/core/saved/test_01.J.out
  head/contrib/libxo/tests/core/saved/test_01.JP.out
  head/contrib/libxo/tests/core/saved/test_01.T.out
  head/contrib/libxo/tests/core/saved/test_01.X.out
  head/contrib/libxo/tests/core/saved/test_01.XP.out
  head/contrib/libxo/tests/core/saved/test_12.J.out
  head/contrib/libxo/tests/core/saved/test_12.JP.out
  head/contrib/libxo/tests/core/test_01.c
  head/contrib/libxo/tests/gettext/saved/gt_01.J.out
  head/contrib/libxo/tests/gettext/saved/gt_01.JP.out
  head/lib/libxo/add.man
  head/lib/libxo/xo_config.h
  head/usr.bin/xohtml/xohtml.sh
Comment 10 commit-hook freebsd_committer 2018-05-31 23:56:28 UTC
A commit references this bug:

Author: gjb
Date: Thu May 31 23:56:00 UTC 2018
New revision: 334458
URL: https://svnweb.freebsd.org/changeset/base/334458

Log:
  MFC r334068 (phil):
   Import libxo-0.9.0:
    - Add xo_format_is_numeric() with improved logic to decide if format
      strings are numeric, so json output quotes them
    - Convert docs to sphinx/rst
    - update tests

  PR:		221676
  Approved by:	re (marius)
  Sponsored by:	The FreeBSD Foundation

Changes:
_U  stable/11/
  stable/11/contrib/libxo/configure.ac
  stable/11/contrib/libxo/doc/Makefile.am
  stable/11/contrib/libxo/doc/_static/
  stable/11/contrib/libxo/doc/_templates/
  stable/11/contrib/libxo/doc/api.rst
  stable/11/contrib/libxo/doc/conf.py
  stable/11/contrib/libxo/doc/example.rst
  stable/11/contrib/libxo/doc/faq.rst
  stable/11/contrib/libxo/doc/field-formatting.rst
  stable/11/contrib/libxo/doc/field-modifiers.rst
  stable/11/contrib/libxo/doc/field-roles.rst
  stable/11/contrib/libxo/doc/format-strings.rst
  stable/11/contrib/libxo/doc/formatting.rst
  stable/11/contrib/libxo/doc/getting.rst
  stable/11/contrib/libxo/doc/howto.rst
  stable/11/contrib/libxo/doc/index.rst
  stable/11/contrib/libxo/doc/intro.rst
  stable/11/contrib/libxo/doc/libxo-manual.html
  stable/11/contrib/libxo/doc/options.rst
  stable/11/contrib/libxo/doc/xo.rst
  stable/11/contrib/libxo/doc/xohtml.rst
  stable/11/contrib/libxo/doc/xolint.rst
  stable/11/contrib/libxo/doc/xopo.rst
  stable/11/contrib/libxo/libxo/libxo.c
  stable/11/contrib/libxo/tests/core/saved/test_01.E.out
  stable/11/contrib/libxo/tests/core/saved/test_01.H.out
  stable/11/contrib/libxo/tests/core/saved/test_01.HIPx.out
  stable/11/contrib/libxo/tests/core/saved/test_01.HP.out
  stable/11/contrib/libxo/tests/core/saved/test_01.J.out
  stable/11/contrib/libxo/tests/core/saved/test_01.JP.out
  stable/11/contrib/libxo/tests/core/saved/test_01.T.out
  stable/11/contrib/libxo/tests/core/saved/test_01.X.out
  stable/11/contrib/libxo/tests/core/saved/test_01.XP.out
  stable/11/contrib/libxo/tests/core/saved/test_12.J.out
  stable/11/contrib/libxo/tests/core/saved/test_12.JP.out
  stable/11/contrib/libxo/tests/core/test_01.c
  stable/11/contrib/libxo/tests/gettext/saved/gt_01.J.out
  stable/11/contrib/libxo/tests/gettext/saved/gt_01.JP.out
  stable/11/lib/libxo/add.man
  stable/11/lib/libxo/xo_config.h