Re: What string encoding to pick as standard for a programming language?

On 21 Jun, 15:25, Marcin 'Qrczak' Kowalczyk <qrc...@xxxxxxxxxx> wrote:
....
Searching of human texts is more complex than choosing a normalization
(case insensitive searches, ignoring diacritics, ignoring differences
between similar but distinct glyphs like «'» and «'», or «-» and «-»).
Choosing a different string representation would not be enough to reduce
it to string comparison.

OTOH there is no point in considering glyph similarity when parsing
keywords from a configuration file, or when comparing filenames.

Agreed.

It's not obvious however how to design the details, there are lots
of possibilities and lots of needs. Let's consider: different rules
for different locales, specifying the Unicode version explicitly or
leaving it unspecified, letting the end user override collation rules,
obtaining behavior which is compatible with particular external tools
(databases, filesystems, programming languages), etc.

And because this is hard and choices are rarely obvious, the core
string handling rules should be simple, so complex algorithms can be
implemented upon them. It's easier to apply some normalization than
to unapply one which has been applied automatically before.

Yes, this seems a very hard thing to design. At the moment I am
thinking to hide whatever implementation behind a standard interface
because I cannot think of there being any direct access that could
possibly apply to all character sets.

As for the actual representation perhaps, as a first attempt, there is
a parallel in the motivations for VLIW processors: different bit
fields represent different characteristics of the characters, e.g. so
many bits for the class of character (punct, letter, numeral, space),
so many for the base character, so many for meaning-changing accents,
so many for accents which do not change the meaning and perhaps affect
only the text when read, so many for variants of the character: case
etc. Sure, each 'character' would be much wider than normal. However,
awkward searches such as for a given accent, for a word that starts
with a given letter, for a piece of text regardless of accents etc.
would be easier to carry out.

Note, characters don't need to be stored in the above form either in
memory or on disk as long as there is a (fast) mapping to the long
form. Hmmm.....

.

Relevant Pages

  • [TOMOYO #15 3/8] Common functions for TOMOYO Linux.
    ... This file contains common functions (e.g. policy I/O, pattern matching). ... Since TOMOYO Linux is a name based access control, ... TOMOYO Linux's string manipulation functions make reviewers feel crazy, ... the Linux kernel accepts all characters but NUL character ...
    (Linux-Kernel)
  • Re: ToUpper() Better solution
    ... Looking into it, I see a simpler method: After normalization, use ... characters (which will be the accents) with the empty string. ... every possible character, where it defaults to the Unicode character, ...
    (microsoft.public.dotnet.general)
  • RfD: Escaped Strings version 4
    ... the S" string can only contain printable characters, ... the S" string cannot contain the '"' character, ... as an escape character for the entry of characters that cannot be ... \b BS (backspace, ASCII 8) ...
    (comp.lang.forth)
  • RfD: Escaped Strings version 4
    ... the S" string can only contain printable characters, ... the S" string cannot contain the '"' character, ... as an escape character for the entry of characters that cannot be ... \b BS (backspace, ASCII 8) ...
    (comp.lang.forth)
  • Re: ToUpper() Better solution
    ... Looking into it, I see a simpler method: After normalization, use ... characters (which will be the accents) with the empty string. ... every possible character, where it defaults to the Unicode ...
    (microsoft.public.dotnet.general)