% Copyright 2016 The LaTeX3 Project
\documentclass{ltnews}

\PassOptionsToPackage{colorlinks}{hyperref}

\usepackage{csquotes}
\usepackage{hologo}
\usepackage{ragged2e}
\usepackage{underscore}

\AtBeginDocument{%
  \renewcommand*{\LaTeXNews}{\LaTeX3~News}%
  \RaggedRight
  \setlength\parindent{1em}%
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\publicationmonth{November}
\publicationyear{2016}
\publicationissue{10}

% Avoid hyphenation of csnames
\makeatletter
\protected\edef\cs#1{%
  \noexpand\path{\@backslashchar#1}%
}
\makeatother

\begin{document}

\maketitle

There has been something of a gap since the last \LaTeX3 News, but this does
not mean that work has not been going on. The Team have been working on a
number of areas, many of which reflect wider take-up of \pkg{expl3}. There have
also been a number of significant new developments in the \LaTeX3
\enquote{sphere} in the last two years.

\section{\pkg{l3build}: Testing \LaTeX{} packages}

Testing has been an important part of the work of the team since they assumed
maintenance of \LaTeX{} over twenty years ago. Various scripts have been used
over that time by the team for testing, but these have until recently not been
set up for wider use.

With the general availability of \hologo{LuaTeX} it is now possible to be sure
that every \TeX{} user has a powerful general scripting language available:
Lua. The team have used this to create a new general testing system for \TeX{}
code, \pkg{l3build}. This \emph{is} designed to be used beyond the team, so is
now available in \TeX{} Live and \hologo{MiKTeX} and is fully documented.
Testing using \pkg{l3build} makes use of a normalised version of the
\texttt{.log} file, so can test any aspect of \TeX{} output (e.g., by using
\cs{showbox}) or its algorithms (by displaying results in the \texttt{.log}).

Part of the remit for creating \pkg{l3build} was to enable the team to work
truly cross-platform and to allow testing using multiple \TeX{} engines
(earlier systems were limited to a single engine, normally \eTeX{}). The new
testing system means we are in a much stronger position to support a variety of
engines (see below). It has also enabled us to give useful feedback on
development of the \hologo{LuaTeX} engine.

As well as the core capability in testing, \pkg{l3build} also provides a
\enquote{one stop} script for creating release bundles. The script is
sufficiently flexible that for many common \LaTeX{} package structures, setting
up for creating releases will require only a few lines of configuration.

In addition to the documentation distributed with \pkg{l3build}, the project
website~\cite[publications in 2014]{10:project-publications} contains some
articles, videos and conference presentations that explain how to use
\pkg{l3build} to manage and test any type of (\LaTeX{}) package.

\section{Automating \pkg{expl3} testing}

As well as developing \pkg{l3build} for local use, the team have also set up
integration testing for \pkg{expl3} using the Travis-CI system. This means that
\emph{every} commit to the \LaTeX3 code base now results in a full set of tests
being run. This has allowed us to significantly reduce the number of occasions
where \pkg{expl3} needs attention before being released to CTAN.

Automated testing has also enabled us to check that \pkg{expl3} updates do not
break a number of key third-party packages which use the programming
environment.

\section{Refining \pkg{expl3}}

Work continues to improve \pkg{expl3} both in scope and robustness. Increased
use of the programming environment means that code which has to-date been
under-explored is being used, and this sometimes requires changes to the code.

The team have extended formal support in \pkg{expl3} to cover the engines
p\TeX{} and up\TeX{}, principally used by Japanese \TeX{} users. This has been
possible in part due to the \pkg{l3build} system discussed above.
Engine-dependent variations between \hologo{pdfTeX}, \hologo{XeTeX},
\hologo{LuaTeX} and (u)p\TeX{} are now well-understood and documented. As part
of this process, the \enquote{low-level} part of \pkg{expl3}, which saves all
primitives, now covers essentially all primitives found in all of these
engines.

The code in \pkg{expl3} is now entirely self-contained, loading no other
third-party packages, and can also be loaded as a generic package with plain
\TeX{}, etc. These changes make it much easier to diagnose problems and
make \pkg{expl3} more useful. In particular it can be used as a programming
language for generic packages, that then can run without modifications under
different formats!

The team have made a range of small refinements to both internals and
\pkg{expl3} interfaces. Internal self-consistency has also been improved, for
example removing almost all use of \texttt{nopar} functions. Performance
enhancements to the \pkg{l3keys} part of \pkg{expl3} are ongoing and should
result in significantly faster key setting. As keyval methods are increasingly
widely used in defining behaviours, this will have an impact on compile times
for end users.

\section{Replacing \cs{lowercase} and \cs{uppercase}}

As discussed in the last \LaTeX3 News, the team have for some time been keen to
provide new interfaces which do not directly expose (or in some cases even use)
the \TeX{} primitives \cs{lowercase} and \cs{uppercase}. We have now created a
series of different interfaces that provide support for the different
conceptual uses which may flow from the primitives:
\begin{itemize}
  \item For case changing text, \cs{tl_upper_case:n}, \cs{tl_lower_case:n},
    \cs{tl_mixed_case:n} and related language-aware functions. These are
    Unicode-capable and designed for working with text. They also allow for
    accents, expansion of stored text and leaving math mode unchanged.  At
    present some of the interface decisions are not finalised so they are
    marked as experimental, but the team expect the core concept to be stable.
  \item For case changing programming strings, \cs{str_upper_case:n},
    \cs{str_lower_case:n} and \cs{str_fold_case:n}. Again these are
    Unicode-aware, but in contrast to the functions for text are not
    context-dependent. They are intended for caseless comparisons, constructing
    command names on-the-fly and so forth.
  \item For creating arbitrary character tokens, \cs{char_generate:nn}. This
    is based on the \cs{Ucharcat} primitive introduced by \hologo{XeTeX}, but
    with the ideas extended to other engines. This function can be used to
    create almost any reasonable token.
  \item For defining active characters, \cs{char_set_active_eq:NN} and
    related functions. The concept here is that active characters should be
    equivalent to some named function, so one does not directly define the
    active character.
\end{itemize}

\section{Extending \pkg{xparse}}

After discussions at TUG2015 and some experimentation, the team have added a
new argument type, \texttt{e} (\enquote{embellishment}), to \pkg{xparse}.
This allows arguments similar to
\TeX{} primitive sub- and superscripts to be accepted. Thus
\begin{verbatim}
\DeclareDocumentCommand\foo{e{^_}}
  {\showtokens{"#1"}}
\foo^{Hello} world
\end{verbatim}
will show
\begin{verbatim}
"{Hello}{-NoValue-}".
\end{verbatim}

At present, this argument type is experimental: there are a number of models
which may make sense for this interface.

\section{A new \cs{parshape} model}

As part of development of \pkg{l3galley}, Joseph Wright has proposed a new
model for splitting up the functions of the \cs{parshape} primitive into three
logical elements:
\begin{itemize}
  \item Margins between the edges of the galley and the paragraph (for example
    an indented block);
  \item Cut-out sections running over a fixed number of lines, to support
    \enquote{in place} figures and so forth;
  \item Running or single-paragraph shape.
\end{itemize}

There are additional elements to consider here, for example whether lines are
the best way to model the length of shaping, how to handle headings, cut-outs
at page breaks, etc.


\section{Globally optimized pagination of documents}

Throughout 2016 Frank Mittelbach has worked on methods and algorithms for
globally optimizing the pagination of documents including those that contain
floats. Early research results have been presented at Bacho\TeX{} 2016, TUG
2016 in Toronto and later in the year at \mbox{DocEng'16}, the ACM Symposium on
Document Engineering in Vienna. A link to the ACM paper (that allows a download
free of charge) can be found on the project
website~\cite{10:project-publications}. The site also holds the speaker notes from
Toronto and will host a link to a video of the presentation once it becomes
available.

The framework developed by Frank is based on the extended functionality
provided by \hologo{LuaTeX}, in particular its callback functions that allow
interacting with the typesetting process at various points. The algorithm that
determines the optimal pagination of a given document is implemented in {Lua}
and its results are then used to direct the formatting done by the \TeX{}
engine.

At the current point in time this a working prototype but not yet anywhere near
a production-ready system. However, the work so far shows great potential and
Frank is fairly confident that it will eventually become a generally usable
solution.

\section{Looking forward}

The \hologo{LuaTeX} engine has recently reached version~1.0. This may presage a
stable \hologo{LuaTeX} and is likely to result in wider use of this engine in
production documents.If that happens we expect to implement some of the more
complex functionality (such as complex pagination requirements and models) only
for \hologo{LuaTeX}.

\begin{thebibliography}{10}
  \raggedright
  \bibitem{10:project-publications}
    Links to various publications by members of the \LaTeX{} Project Team.
    \newblock \url{https://www.latex-project.org/publications}.
\end{thebibliography}

\end{document}
