Bug 203040 - Nvi truncates files with non-ASCII characters
Summary: Nvi truncates files with non-ASCII characters
Status: Closed DUPLICATE of bug 202290
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 10.2-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-09-11 12:45 UTC by Bjorn Robertsson
Modified: 2016-01-15 13:50 UTC (History)
0 users

See Also:


Attachments
ISO-8859 file containing ;ð;; string, which Nvi will not load (6 bytes, text/plain)
2015-09-11 12:45 UTC, Bjorn Robertsson
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Bjorn Robertsson 2015-09-11 12:45:09 UTC
Created attachment 160929 [details]
ISO-8859 file containing ;ð;; string, which Nvi will not load

Nvi 2.1.2 (from FreeBSD 10.0 to FreeBSD 10.2) displays this behaviour in our system, when a certain character composition appears in a file - the file will be truncated from that line, with a false EOF placed in the line, making the file unusable.

A file with non-ASCII characters (like eth) can be created to verify the problem:
;ð;;

(the file should be ISO8859, the login.conf has lang, charset and lc_all set to iso-8859-1, added to default:
        :charset=ISO8859-1:\
        :lang=is_IS.ISO8859-1:\
        :lc_all=is_IS.ISO8859-1: \
The client terminal is set to ISO or Western-1252).

A file, which contains only
;ð;
will not be truncated.

After saving and re-opening the file, the fileencoding will be set to utf-8, by Nvi.

vi will display file with multiple lines like after re-opening:
line 1
line 2
line 3
~
line 5

line 4 was ;ð;; when the file was created, and after re-opening the file content after ~ is unusable. Other strings like ð ; will also break. Not all non-ASCII characters will do this, ý will be ok, but á will not... (y acute, a acute)

Replacing vi with Nvi 1.79 from FreeBSD 9 does alleviate the problem.

By unsetting login.conf to defaults, a file can be created, saved and re-opened, but it will not display the characters correctly, it will look like:
;\xf0;;
instead of
;ð;;

Setting LANG will again break Nvi with the message:
Conversion error on line 1; FILE: unmodified: line 1

I've described the problem to Sven Verdoolaege also.

Currently being very careful, or installing Nvi 1.79 from FreeBSD 9.3 source tree.
Comment 1 Bjorn Robertsson 2015-12-17 11:12:57 UTC
Also, using nano from ports to view these files is successful.

Of course ð is not the only character - ;å;; for example behaves the same.
;ð; ;ð;; <- bad
;ð ;ð;; <- ok

This *must* affect other people
Comment 2 Bjorn Robertsson 2016-01-15 13:50:08 UTC
Tested patch from https://bugs.freebsd.org/bugzilla/attachment.cgi?id=165487&action=diff

And it seems to have resolved this problem

*** This bug has been marked as a duplicate of bug 202290 ***