Bug 197499

Summary: wc libxo emits invalid JSON when used on multiple files
Product: Base System Reporter: Allan Jude <allanjude>
Component: binAssignee: Marcel Moolenaar <marcel>
Status: Closed FIXED    
Severity: Affects Some People CC: marcel
Priority: ---    
Version: CURRENT   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
Patch for wc that puts the total object outside the file array
none
removes the 'total' key, making the total just a regular array element
none
make 'total' the last element of the file array [updated patch] none

Description Allan Jude freebsd_committer freebsd_triage 2015-02-09 20:31:10 UTC
When using wc's libxo JSON output method, it emits invalid JSON.

Steps to recreate:

wc --libxo json /etc/defaults/*



The output is (pretty-printed for readability):



{
  "wc": {
    "file": [
      {
        "lines": 111,
        "words": 644,
        "characters": 3999,
        "filename": "/etc/defaults/bluetooth.device.conf"
      },
      {
        "lines": 87,
        "words": 403,
        "characters": 2439,
        "filename": "/etc/defaults/devfs.rules"
      },
      {
        "lines": 385,
        "words": 1228,
        "characters": 11700,
        "filename": "/etc/defaults/periodic.conf"
      },
      {
        "lines": 692,
        "words": 4576,
        "characters": 36171,
        "filename": "/etc/defaults/rc.conf"
      },
      "total": {
        "lines": 1275,
        "words": 6851,
        "characters": 54309,
        "filename": "total"
      }
    ]
  }
}


The problem is that the 'total' segment, is a key/value pair, but that is not allowed inside an array.

The output would either need to be changed to remove the key (the other elements in the array use the filename for identification), or be made a sibling of the 'file' 2nd level element, like so:

{
  "wc": {
    "file": [
      {
        "lines": 111,
        "words": 644,
        "characters": 3999,
        "filename": "/etc/defaults/bluetooth.device.conf"
      },
      {
        "lines": 87,
        "words": 403,
        "characters": 2439,
        "filename": "/etc/defaults/devfs.rules"
      },
      {
        "lines": 385,
        "words": 1228,
        "characters": 11700,
        "filename": "/etc/defaults/periodic.conf"
      },
      {
        "lines": 692,
        "words": 4576,
        "characters": 36171,
        "filename": "/etc/defaults/rc.conf"
      }
    ],
    "total": {
      "lines": 1275,
      "words": 6851,
      "characters": 54309,
      "filename": "total"
    }
  }
}
Comment 1 Allan Jude freebsd_committer freebsd_triage 2015-02-09 20:59:00 UTC
I am not clear which was the intended output, so I have attached a patch for each way:

'inside' (total is the last element on the file array)

and 'outside' (total is a sibling key, at the same level as the file array)
Comment 2 Allan Jude freebsd_committer freebsd_triage 2015-02-09 21:01:54 UTC
Created attachment 152820 [details]
Patch for wc that puts the total object outside the file array
Comment 3 Allan Jude freebsd_committer freebsd_triage 2015-02-09 21:02:22 UTC
Created attachment 152821 [details]
removes the 'total' key, making the total just a regular array element
Comment 4 Allan Jude freebsd_committer freebsd_triage 2015-02-10 00:06:18 UTC
Created attachment 152825 [details]
make 'total' the last element of the file array [updated patch]
Comment 5 Marcel Moolenaar freebsd_committer freebsd_triage 2015-02-10 20:33:50 UTC
I don't seem to be able to reproduce your problem:

fbsdvm64% wc --libxo json,pretty /etc/defaults/*
{
  "wc": {
    "file": [
      {
        "lines": 111,
        "words": 644,
        "characters": 3999,
        "filename": "/etc/defaults/bluetooth.device.conf"
      },
      {
        "lines": 87,
        "words": 403,
        "characters": 2439,
        "filename": "/etc/defaults/devfs.rules"
      },
      {
        "lines": 381,
        "words": 1221,
        "characters": 11568,
        "filename": "/etc/defaults/periodic.conf"
      },
      {
        "lines": 699,
        "words": 4593,
        "characters": 36522,
        "filename": "/etc/defaults/rc.conf"
      }
    ],
    "total": {
      "lines": 1278,
      "words": 6861,
      "characters": 54528,
      "filename": "total"
    }
  }
}
fbsdvm64% 

"file" is a proper array and "total" falls outside of the array. jsonlint.com flags this as valid JSON.

I'm running sources as of today.

Do you have older sources or modified sources?
Comment 6 Allan Jude freebsd_committer freebsd_triage 2015-02-11 05:17:04 UTC
(In reply to Marcel Moolenaar from comment #5)

My results are different:
# uname -a
FreeBSD Nexus.HML3.ScaleEngine.net 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r276776: Wed Jan  7 16:16:14 UTC 2015     root@Nexus.HML3.ScaleEngine.net:/usr/obj/usr/src/sys/NEXUS  amd64


# wc --libxo json,pretty /etc/defaults/*
{
  "wc": {
    "file": [
      {
        "lines": 111,
        "words": 644,
        "characters": 3999,
        "filename": "/etc/defaults/bluetooth.device.conf"
      },
      {
        "lines": 87,
        "words": 403,
        "characters": 2439,
        "filename": "/etc/defaults/devfs.rules"
      },
      {
        "lines": 385,
        "words": 1228,
        "characters": 11700,
        "filename": "/etc/defaults/periodic.conf"
      },
      {
        "lines": 692,
        "words": 4576,
        "characters": 36171,
        "filename": "/etc/defaults/rc.conf"
      },
      "total": {
        "lines": 1275,
        "words": 6851,
        "characters": 54309,
        "filename": "total"
      }
    ]
  }
}


Let me try a newer build
Comment 7 Marcel Moolenaar freebsd_committer freebsd_triage 2015-02-11 17:57:06 UTC
Committed revision 278590.
Comment 8 commit-hook freebsd_committer freebsd_triage 2015-02-11 17:57:13 UTC
A commit references this bug:

Author: marcel
Date: Wed Feb 11 17:56:25 UTC 2015
New revision: 278590
URL: https://svnweb.freebsd.org/changeset/base/278590

Log:
  Close the file list before opening the container that holds the
  totals, otherwise we end up emitting invalid JSON -- provided
  libxo does not prevent us from doing that.

  PR:		197499
  Submitted by:	allanjude@

Changes:
  head/usr.bin/wc/wc.c