@Chapter
    @Title { Lexical structure }
    @Tag { lexical }
@Begin
Nonpareil source files are interpreted as sequences of {@I tokens}:
token @Index { Token }
@ID @Tbl
    aformat { @Cell A | @Cell B }
    mv { 0.5vx }
{
@Rowa
    font { Italic }
    A { Token class }
    B { Example }
    rb { yes }
@Rowa
    A { Identifiers }
    B { @Nonpareil { list } }
@Rowa
    A { Reserved words }
    B { @Nonpareil { else } }
@Rowa
    A { User-defined punctuation sequences }
    B { @Nonpareil { + } }
@Rowa
    A { Reserved punctuation sequences }
    B { @Nonpareil { := } }
@Rowa
    A { Boolean literals }
    B { @Nonpareil { false } }
@Rowa
    A { Number literals }
    B { @Nonpareil { 3.1416 } }
@Rowa
    A { Character literals }
    B { @Nonpareil { 'H' } }
@Rowa
    A { String literals }
    B { @Nonpareil { "Hello" } }
    rb { yes }
}
Tokens are separated by white space, comments, or nothing.  As a file
is read, the longest legal token at each point is always chosen.
This chapter defines these eight token classes precisely.
@BeginSections

@Include { lexical.charset }
@Include { lexical.whitespace }
@Include { lexical.identifiers }
@Include { lexical.punctuation }
@Include { lexical.literals }

@EndSections
@End @Chapter
