@Chapter
    @PartNumber { Part A }
    @PartTitle { The Platform }
    @Title { Introduction }
    @Tag { intro }
@Begin
Some instances of high school timetabling problems, taken from
institutions in several countries and specified formally in an XML
format called XHSTT, have recently become available @Cite { $post2008 }.
For the first time, the high school timetabling problem can be studied
in its full generality.
@PP
KHE is an open-source ANSI C library, released under the GNU public
licence, which aims to provide a fast and robust foundation for solving
instances of high school timetabling problems expressed in the XHSTT
format.  Users of KHE may read and write XML files, create solutions,
and add and change time and resource assignments using any algorithms
they wish.  The cost of the current solution is always available,
kept up to date by a hand-coded constraint propagation network.
KHE also offers features inherited from the author's KTS system
@Cite { $kingston2006kts, $kingston2007kts_paper }, notably layer
trees and matchings, and solvers for several major sub-tasks.
@PP
KHE is intended for production use, but it is also a research
vehicle, so new versions will not be constrained by backward
compatibility.  Please report bugs to me at
@I { jeff@it.usyd.edu.au }.  I will release a corrected version
within a few days of receiving a bug report, wherever possible.
@PP
This introductory chapter explains how to install and use KHE,
surveys its data types, and describes some operations common
to many types.
@BeginSections

@Section
    @Title { Installation and use }
    @Tag { intro.install }
@Begin
@LP
KHE has a home page, at
@ID @F @Verbatim { http://jeffreykingston.id.au/khe/ }
The current version of KHE is a gzipped tar file in that directory.
The current version of this documentation (a PDF file) is also
stored there.  The names of these files change with each release;
they are most easily downloaded using links on the home page.
@PP
Originally, `KHE' stood for `Kingston's High School Timetabling
Engine', but it now covers all timetabling software released by
me:  the platform, the solvers, HSeval (which drives the HSEval
web site), my nurse rostering software, and anything else I
release in the future.  So `KHE' no longer stands for anything,
except possibly `Kingston's Humungous Enterprise'.
@PP
I have used different kinds of version numbers over the years,
but starting with Version 2.1 I am reverting to the traditional
form, of a major release number and minor release number separated
by a dot.  Each KHE release is a release of all my software under
a single version number.
@PP
A program that incorporates the KHE platform can gain access to the
current version number by calling
@ID @C {
char *KheVersionNumber(void);
char *KheVersionBanner(void);
}
For example, if Version 2.6 is compiled into the program that
calls these functions, their results will be @C { "2.6" } and
@C { "Version 2.6 (March 2021)" }.
@PP
To install KHE, download a release and unpack it using @C { gunzip }
and @C { tar xf } as usual.  The resulting directory, @C { khe },
contains a makefile, some @C { src_* } directories holding the source
files of KHE, and some @C { doc_* } directories holding the source
files of this documentation.  Consult the makefile for information
about what's what in the distribution, and how to install and use KHE.
@PP
Starting with Version 2.1, the KHE source files are divided into three
parts:  the platform (whose interface is file @C { khe_platform.h }),
the solvers (@C { khe_solvers.h }), and a main program.  This allows
users to use only the platform, or it and the solvers, or those plus
a main program.  The distribution also contains two source directories
holding my nurse rostering software.
#@PP
#KHE uses Posix threads to implement solving in parallel (function
#@C { KheParallelSolve } from Section {@NumberOf general_solvers.parallel}),
#so the compiler and linker commands need the @C { -pthread } flag.  If
#you don't have Posix threads, the makefile documents a workaround.
#The only difference will be that parallel solvers will do their
#solving sequentially rather than in parallel.
#@PP
#Another possible porting problem arises in those parts of KHE which
#consult the system to find out how much time has been consumed while
#solving.  Again, there is a workaround for this in the makefile,
#which if taken will cause all time measurements to be 0.
@End @Section

@Section
    @Title { The data types of KHE }
    @Tag { intro.data }
@Begin
@LP
This section is an overview of KHE's data types.  The following
chapters have the details.
@PP
Type @C { KHE_ARCHIVE } represents one archive, that is, a collection of
instances plus a collection of solution groups.  Type @C { KHE_SOLN_GROUP }
represents one solution group, that is, a set of solutions of the
instances of the archive it lies in.  The word `solution' is abbreviated
to `soln' wherever it appears in the KHE interface.  Use of these types
is optional:  instances do not have to lie in archives, and solutions
do not have to lie in solution groups.
@PP
Type @C { KHE_INSTANCE } represents one instance of the high school
timetabling problem.  @C { KHE_TIME_GROUP } represents a set of times;
@C { KHE_TIME } represents one time.  @C { KHE_RESOURCE_TYPE }
represents a resource type (typically @I { Teacher }, @I { Room },
@I { Class }, or @I { Student }); @C { KHE_RESOURCE_GROUP } represents
a set of resources of one type; and @C { KHE_RESOURCE } represents
one resource.
@PP
Type @C { KHE_EVENT_GROUP } represents a set of events; @C { KHE_EVENT }
represents one event, including all information about its time.  Type
@C { KHE_EVENT_RESOURCE } represents one resource element within an
event.  Type @C { KHE_CONSTRAINT } represents one constraint.  This
could have any of the constraint types of the XML format (it is their
abstract supertype).
@PP
Type @C { KHE_SOLN } represents one solution, complete or partial,
of a given instance, optionally lying within a solution group.  Type
@C { KHE_MEET } represents one meet (KHE's commendably brief name
for what the XML format calls a solution event, split event, or
sub-event:  one event as it appears in a solution), including all
information about its time.  Type @C { KHE_TASK } represents one
piece of work for a resource to do:  one resource element within a meet.
@PP
KHE supports multi-threading by ensuring that each instance and its
components (of type @C { KHE_INSTANCE }, @C { KHE_TIME_GROUP }, and
so on) is immutable after loading of the instance is completed, and
that operations applied to one solution object do not interfere with
operations applied simultaneously to another.
#@FootNote {
#Assuming that KHE is linked to an implementation of @C { malloc() }
#suited to multiple threads, such as the Linux @C { glibc }
#implementation by Doug Lea and W.~Gloger.  KHE does not leak
#memory, although, since garbage collection is not standard in C,
#the user must indicate when major objects, such as instances
#and solutions, are no longer required.
#}
Thus, after instance loading is completed, it is safe to create
multiple threads with different @C { KHE_SOLN } objects in each
thread, all referring to the same instance, and operate on those
solutions in parallel.  No such guarantees are given for operating
on the same solution from different threads.
@End @Section

@Section
    @Title { Memory allocation using arenas and arena sets }
    @Tag { intro.arena }
@Begin
@LP
Large solves of timetabling instances can make heavy demands on
memory allocators.  A search of the internet will show that there
is no ideal memory allocator, that is, one that implements the
C @C { malloc }, @C { realloc }, and @C { free } functions with
negligible time and memory overhead, taking the demands of memory
caching and multi-threading into account.  What follows is an attempt
to do memory allocation for KHE well at the cost of requiring the
user to pay more attention to it than is customary.  For serious
solving this seems to the author to be the right tradeoff.
@PP
An @I { arena memory allocator } allocates memory from an area
called an @I { arena }.  A KHE arena enlarges, effectively without
limit, as memory from it is allocated.  There is no operation to
free one piece of arena memory.  Instead, the entire arena is
freed in one operation.  This allows an arena memory allocator to
be faster than a general memory allocator, and to use less memory.
@PP
An arena memory allocator is used throughout KHE.  It is flown in
from another project of the author's called Howard, so the type
name of a memory arena is not @C { KHE_ARENA } but @C { HA_ARENA }.
One can create an arena, obtain memory from it, and delete (free)
it.  The memory can have a fixed size, or it can be resizable
(supporting varying-length arrays and hash tables).
@PP
Every arena belongs to exactly one @I { arena set }, an object
of type @C { HA_ARENA_SET } representing a set of arenas.
When an arena has no memory available to satisfy a request for
memory from a caller, it obtains a new chunk of memory from its
arena set; and when an arena is deleted, its chunks return to
its arena set and become available to other arenas belonging
to that arena set.
@PP
There is no locking within types @C { HA_ARENA } and
@C { HA_ARENA_SET }, other than within the underlying calls
to @C { malloc } which are the ultimate source of the memory
that arena sets give out to their arenas.  In a multi-threaded
program, this constrains how arenas and arena sets are used.
@PP
The rule for arena sets is very simple.  There should be exactly
one arena set per thread, and all memory allocated by code
running within that thread should come from arenas that belong
to that arena set.  This ensures that no arena set or arena is
ever accessed by two different threads, and so no locking of
arena sets or arenas is required.  (There is no prohibition on
having two arena sets within one thread, but that is not
recommended because it can waste memory.)
@PP
When a new thread is begun, a new arena set should be created
and passed to all code in that thread that needs memory.  When
a thread terminates, the operation for merging one arena set
into another should be called from the parent thread to merge
the terminating child thread's arena set into the parent
thread's arena set.  All variables pointing to the child thread's
arena set must be changed to point to the parent thread's arena
set.  In practice users do not have to worry about all this,
because it is done for them by @C { KheArchiveParallelSolve }
(Section {@NumberOf general_solvers.parallel}).
@PP
When some single-threaded code needs memory, in simple cases it
can be passed a single arena, from which it can obtain its memory.
That memory remains in use until the caller who passed it the
arena deletes that arena.  In complex cases the code should be
passed the arena set of its thread, so that it can create and
delete its own arenas as required.
@PP
For example, a solution object, of type @C { KHE_SOLN }, is a
large, complex object.  It is created by function @C { KheSolnMake }
(Section {@NumberOf solutions.top}) in its own arena, so that it
can be deleted individually, if required, by deleting the arena.
But there are also @I { placeholder solutions } which are mostly
deleted but which retain a small amount of information, including
solution cost.  So in fact a solution is created in two arenas,
one holding the placeholder part and the other holding the rest.
Reducing a full solution to a placeholder involves deleting the
second arena but not the first.  Given this complexity, the right
course is to pass the enclosing thread's arena set to
@C { KheSolnMake }, and leave it to @C { KheSolnMake } to create
whatever arenas it needs.
@PP
Similar considerations apply to solvers created by users.
Typically, a solver will create an arena, use it to hold
whatever objects it uses, and then, when its solve operation
is complete, it will delete its arena, freeing up the memory
it used, and leaving no trace of itself other than any changes
it may have made to the solution object it worked on.  For
this it needs to be passed its thread's arena set.  But in
simple cases it is also reasonable for a solver to accept
an arena parameter and simply use that arena.
@PP
KHE supports multi-threading, but it assumes that each solution
object will be accessed by only one thread.  Each solution
contains the arena set of that thread, and a solver can
retrieve that arena set and use it to create arenas.
Alternatively, a solver can call @C { KheSolnArenaBegin }
(Section {@NumberOf solutions.top.arenas}) to obtain one arena
from that arena set, and @C { KheSolnArenaEnd } to delete that
arena when it is no longer required.  In practice this is
the usual way to gain access to an arena.
@PP
When solutions are read single-threaded for evaluation, they
all reference the same arena set.  When they are created
for solving in parallel, they have different arena sets.
There is no simple way to transition from one of these
situations to the other.
@PP
Arena sets offer a means of failing gracefully if memory runs out
(quite likely during large solves).  One can pass a C @C { longjmp }
environment to an arena set, and then if any request for memory
from any of that arena set's arenas cannot be satisfied, the
arena code will long jump to the target of the @C { longjmp }
environment instead of returning.  There is also a simple way
to prevent any one thread from monopolizing the available
memory.  For more on these issues and on arenas and arena
sets generally, consult Appendix {@NumberOf ha}.
@End @Section

@Section
    @Title { Common operations  }
    @Tag { intro.common }
@Begin
@LP
This section describes some miscellaneous operations that are
common to many data types.
# @BeginSubSections
# 
# @SubSection
#     @Title { Back pointers  }
#     @Tag { backptrs }
# @Begin
# @LP
@PP
Whenever KHE creates an object, any string-valued attributes
of that object passed by the user are not stored directly;
instead, malloced copies are stored.  If the object is later
deleted, the malloced copy is deleted along with it.  Thus,
whatever its origin, a string-valued attribute has the same
lifetime as the object itself.
@PP
Use of KHE often involves creating objects that contain references
to KHE entities (objects of types defined by KHE) alongside other
information.  Sometimes it is necessary to go backwards, from
a KHE entity to a user-defined object.  Accordingly, each KHE
entity contains a @I { back pointer } which the user is free to
set and retrieve, using calls which look generically like this:
@ID @C {
void KheEntitySetBack(KHE_ENTITY entity, void *back);
void *KheEntityBack(KHE_ENTITY entity);
}
All back pointers are initialized to @C { NULL }.  In general, KHE
itself does not set back pointers.  The exception is that some
solvers packaged with KHE set the back pointers of the solution
entities they deal with.  This is documented where it occurs.
# In general,
# solvers should assume that all back pointers are undefined when
# they begin, and that they are free to use them.
# @End @SubSection
# 
# @SubSection
#     @Title { Similarity  }
#     @Tag { similarity }
# @Begin
# @LP
@PP
Timetables often contain symmetries of various kinds.  In high
school timetabling, the student group resources of one form are
often symmmetrical:  they attend the same kinds of events over
the course of the cycle.
@PP
Knowledge of similarity can be useful when solving.  For example,
it might be useful to timetable similar events attended by student
group resources of the same form at the same time.  Accordingly,
several KHE entities offer an operation of the form
@ID @C {
bool KheEntitySimilar(KHE_ENTITY e1, KHE_ENTITY e2);
}
which returns @C { true } if KHE considers that the two entities
are similar.  If they are the exact same entity, they are always
considered similar.  In other cases, the definition of similarity
varies with the kind of entity, although it follows a common pattern:
evidence both in favour of similarity and against it is accumulated,
and there needs to be a significant amount of evidence in favour,
and more evidence in favour than against.  For example, an event
containing no event resources will never be considered similar to
any event except itself, since positive evidence, such as requests
for the same kinds of teachers, is lacking.
@PP
Similarity is not a transitive relation in general.  In other words,
if @C { e1 } and @C { e2 } are similar, and @C { e2 } and @C { e3 }
are similar, that does not imply that @C { e1 } and @C { e3 } are
similar.  There is a heuristic aspect to it that seems inevitable,
although the intention is to stay on the safe side:  to declare
two entities to be similar only when they clearly are similar.
# @End @SubSection

#@SubSection
#    @Title { Priority }
#    @Tag { priority }
#@Begin
#@LP
#Many heuristics do the most difficult thing first:  they pack the
#largest item first, colour the node of largest degree first, and
#so on.  To assist with this, functions typified by
#@ID @C {
#int KheEntityPriority(KHE_ENTITY entity);
#}
#are required.  However, there are usually several ways to prioritize
#entities, leading to a need to pass priority functions as parameters.
#So KHE defines `prioritizer' types:
#@ID @C {
#typedef int (*KHE_ENTITY_PRIORITIZER)(KHE_ENTITY entity);
#}
#and its solvers take parameters of these types.  It is conventional
#in KHE for larger values (rather than smaller ones) to indicate
#higher priority, since this is usually what is wanted, as in the
#bin packing and graph colouring examples.  It is not difficult to
#encode other kinds of priorities as integers.  For example, to
#give smaller values priority, simply negate the values returned.
#Multi-level priorities may be encoded as linear combinations.
#@End @SubSection

# @SubSection
#     @Title { Debug prints  }
#     @Tag { debug }
# @Begin
@PP
Another operation that applies to many entities, albeit a humble
one, is printing the current state of the entity as an aid to
debugging.  The KHE operations for this mostly take the form
@ID @C {
void KheEntityDebug(KHE_ENTITY entity, int verbosity,
  int indent, FILE *fp);
}
They produce a debug print of @C { entity } onto @C { fp }.
@PP
The @C { verbosity } parameter controls how much detail is printed.
Any value is acceptable.  A zero or negative value always prints
nothing.  Every positive value prints something, and as the value
increases, more detail is printed, depending, naturally, on the
kind of entity.  Value 1 tries to print the minimum amount of
information needed to identify the entity, often just its name.
@PP
If @C { indent } is non-negative, a multi-line format is used
in which each line begins with at least @C { indent } spaces.
If @C { indent } is negative, the print appears on one line with
no indent and no concluding newline.  Since space is limited,
verbosity may be reduced when @C { indent } is negative.
@PP
Many entities are organized hierarchically.  Depending on the
verbosity, printing an entity may include printing its descendants.
Their debug functions are passed a value for @C { indent } which
is 2 larger than the value received (when non-negative), so that
the hierarchy is represented in the debug output by indenting.
The debug print of one entity usually begins with @F "[" and ends
with a matching {@F "]"}, making it easy to move around the printed
hierarchy using a text editor.
# @End @SubSection
# 
# @EndSubSections
@End @Section

@Section
    @Title { KHE for employee scheduling }
    @Tag { intro.employee }
@Begin
@LP
Recent versions of KHE support the employee scheduling data format
XESTT as well as the high school timetabling format XHSTT.  XESTT
is the same as XHSTT except for a few extensions, which are
documented on the HSEval web site.
@PP
KHE knows whether it is dealing with XESTT or XHSTT, but it
does not care---it supports XESTT, which includes supporting
XHSTT.  When using KHE for high school timetabling, several
parameters of KHE functions have to be given values that
indicate that the extensions available in XESTT are not used.
This mainly affects the operations for creating cluster
busy times and limit busy times constraints.
@End @Section

@EndSections
@End @Chapter
