@Chapter
    @PartNumber { Part B }
    @PartTitle { The NRConv Executable }
    @Title { NRConv and its Converters }
    @Tag { source }
@Begin
This chapter describes NRConv and its converters, passing silently
over routine things.  The material on each converter assumes that the
reader has a detailed knowledge of the source model being converted.
For NRConv usage information, type @F nrconv with no arguments.
@PP
As a rough guide to the complexity of the code, here are line counts
for the C source files:
@CD @Tbl
    aformat { @Cell ml { 0i } A | @Cell mr { 0i } B }
    mv { 0.5vx }
{
@Rowa
    A { @I Lines }
    B { @I Files }
    rb { yes }
@Rowa
    A { 5092 }
    B { @F { coi.c }, @F { coi_cover.c }, and @F { coi_limit.c } }
@Rowa
    A { 1669 }
    B { @F { inrc1.c } }
@Rowa
    A { 1464 }
    B { @F { inrc2.c } }
@Rowa
    A { 1836 }
    B { @F { cq14.c } }
    rb { yes }
}
The format of the Curtois original instances is considerably more
complex than the others.
@BeginSections

@Section
    @Title { Instance models and solution models }
    @Tag { source.models }
@Begin
@LP
NRConv has @I { instance models } and @I { solution models }, which
define the format of source instances and solutions.  Whenever it reads
a source instance or solution file, it has already been informed (via
the @F -i and @F -s command-line flags) which model the file follows.
@PP
An instance model is defined by a text file called an
@I { instance model file }.  For example:
@ID @F @Verbatim {
InstanceSourceFormat: inrc1.xml
Contributor: The organizers of INRC1
Date:
Country:
Description:
Remarks: converted from INRC1 format by NRConv
}
Each line consists of an identifier followed by a colon followed
by any number of spaces followed by an optional value.  The lines
must appear in the order shown.
@PP
@F { InstanceSourceFormat } gives the name of the format in which the
source instances which follow this model are expressed.  There is a
fixed set of valid names, which at present is
@ID @F { coi.xml inrc1.xml inrc2.xml cq14.txt }
but which is easy to expand (see the top of file @C { ins_model.c }).
There is nothing to prevent different instance models from using the
same source format.
@PP
The remaining lines give values for the metadata fields of each
instance.  If there is no date, as above, then the date that
NRConv runs is substituted.  All these fields are to be taken
as default values.  If an instance contains more informative
metadata values, they may replace these ones.
@PP
A solution model is defined by a text file called a
@I { solution model file }.  For example:
@ID @F @Verbatim {
SolnSourceFormat: inrc1-soln.xml
LinkageToInstance: internal
LinkageToSolnGroup: first
Keep: best
SolnGroup: GOAL
Contributor: The GOAL team
Date:
Description:
Publication: http://www.goal.ufop.br/nrp/
Remarks: converted from INRC1 format by NRConv
}
As before, each line consists of an identifier followed by a colon
followed by any number of spaces followed by an optional value, and
the lines must appear in the order shown.
@PP
The @C { SolnSourceFormat }
line gives the name of the format in which the source solutions are
expressed.  There is a fixed set of valid names, which at present is
@ID @F { coi-soln.xml inrc1-soln.xml inrc2-soln.xml cq14-soln.xml }
but which is easy to expand (see the top of file @C { soln_model.c }).
There is nothing to prevent different solution models from using
the same source format.
@PP
The @F { LinkageToInstance } line defines how to determine which
instance a solution is for.  Its value may be @F { internal },
meaning that the solution file contains this information, or
@F { external }, meaning that it doesn't.  In the latter case,
the longer form of the @F { -s } command line flag must be
used to supply this information.
@PP
The @F { LinkageToSolnGroups } line defines how to determine
which solution group a solution should go into.  There is
a fixed set of values for this, which at present is
@ID @F { first cq14 }
but which could expand in the future.  Value @F { first } places
each solution into the first solution group defined in this file
(which would usually be the only one), while @F { cq14 } uses
a complex rule based on the values of the @F { <Algorithm> }
and @F { <CpuTime> } elements of the solution files.
@PP
The @F { Keep } line says how many solutions to keep for each instance
in each solution group.  Acceptable values are @F { all } to keep all
solutions, and @F { best } to keep only one solution, the best.
@PP
The @F { SolnGroup } line defines a solution group with the given
name, which will be added to the archive.  The following lines
define its metadata fields.  They must all be present, in the order
shown.  If there is already a solution group with the given name,
it is not added again, but NRConv checks that it has the same
metadata values as the ones given here, and aborts if not.
@PP
To include multiple solution groups, start again after
@F { Remarks } with another @F { SolnGroup } line, and so on.
Every solution goes into exactly one solution group, so if
there are any solutions at all, there must be at least one
solution group.
@End @Section

@Section
    @Title { The Curtois original instances }
    @Tag { source.coi }
@Begin
@LP
Curtois pioneered the assembly of instances from around the world,
and their expression in a common format.  Published at
@ID @F "http://www.cs.nott.ac.uk/~psztc/NRP/"
under the heading `Original instances', there are 28 of these
instances, with 66 solutions.  Following Curtois, we omit instance
@F { HED01b } (it is very similar to instance @F { HED01 }) and its
solution, so the converter converts 27 instances and 65 solutions,
to archive @F { COI }, using instance source format @F { coi.xml }
and solution source format @C { coi-soln.xml }, which are implemented
by functions in NRConv source files @F { coi.c }, @F { coi_limit.c },
and @F { coi_cover.c }.
@PP
Instances @F { ERMGH.ros }, @F { CHILD.ros }, @F { ERRVH.ros }, and
@F { MER.ros } use the @F { <TimePeriod> } feature, which gives cover
requirements for periods of the day rather than for each shift type.
Several other instances contain multiple cover requirements (for
particular skills) which apply to the same shifts.  Accordingly,
for all the Curtois original instances generally, NRConv produces
a generous number of extra event resources (the exact number is
documented in @F { coi_cover.c }), and constrains them using limit
resources constraints.
@PP
The exception is shifts for which the @F { <AutoAllocate> } attribute
is @C { false }, meaning that the shift can only be preassigned, not
assigned by a solver.  Each such shift is given the exact number of
event resources required to hold the preassignments, and these event
resources are preassigned in the generated instance.
@PP
All instances use @I { pattern constraints } (element @F { <Match> }),
which place minimum and maximum limits on the number of occurrences of
the elements of an arbitrary set of patterns.  These are not convertible
in general.  NRConv analyses them into three convertible cases, and
omits instances with constraints outside these cases (none of the 27
instances is omitted).
@PP
@I { Case 1 }.  The pattern constraint has maximum limit 0 but is
otherwise arbitrary.  Then the patterns within this constraint are
unwanted patterns and are handled as such.
@PP
@I { Case 2 }.  Each pattern of the constraint either contains a
single term, or a sequence of terms all containing @C { 0 }, or
it is one of the last three patterns, and these together match busy
weekends, as in @F { Sat:[123][123] }, @F { Sat:[0][123] }, and
@F { Sat:[123][0] }, assuming three shifts per day.  The constraint
is otherwise arbitrary.  It is converted to a resource contraint
whose minimum and maximum limits are those of the pattern constraint,
and whose time groups express its terms.
@PP
A pattern containing a single term is easily expressible using
one time group for each starting day.  For example, @F { [12] }
is converted to
@ID lines @Break @F {
"{"1Mon1, 1Mon2"}"
"{"1Tue1, 1Tue2"}"
"{"1Wed1, 1Wed2"}"
...
}
and these time groups are added to the resource constraint.
@PP
A pattern whose terms all contain @F { 0 } is converted to one
negative time group for each starting day.  Consider limiting
the number of free weekends, counting a Friday night shift
(shift @F { 3 }) as part of the following weekend.  The pattern is
@F { Fri:[012][0][0] }.  The time groups are
@ID lines @Break @F {
"{"1Fri3, 1Sat1, 1Sat2, 1Sat3, 1Sun1, 1Sun2, 1Sun3"}"*
"{"2Fri3, 2Sat1, 2Sat2, 2Sat3, 2Sun1, 2Sun2, 2Sun3"}"*
...
}
Each time group contains the complement of each term, on successive days.
@PP
Patterns which match busy weekends are easily represented by
positive time groups
@ID lines @Break @F {
"{"1Sat1, 1Sat2, 1Sat3, 1Sun1, 1Sun2, 1Sun3"}"
"{"2Sat1, 2Sat2, 2Sat3, 2Sun1, 2Sun2, 2Sun3"}"
}
and so on.  NRConv looks for this exact case; it does not attempt
to generalize it in any way.
@PP
@I { Case 3 }.  NRConv is hard-wired to generate certain XESTT
constraints when it reaches certain file positions.  This allows
it to handle source constraints that fall outside the cases above
but which are nevertheless convertible.  For example, the pattern
constraints at lines 107--148 of @F { ERMGH.ros } penalize cases of
two consecutive busy weekends.  The instance begins on a Sunday and
ends on a Saturday, and the constraint for the last two weekends does
not fit the cases given above, so NRConv generates a hard-wired limit
active intervals constraint for the whole set.  As it turns out,
every hard-wired case except the one for instance @F { HED01.ros }
described below concerns limiting the number of consecutive busy
weekends, to one, two, or three.
@PP
An awkward pattern constraint occurs at line 95 of instance
@F { ORTEC02.ros }.  The problem patterns, @F { Sat:[NEDL][NEDL] },
@F { Sat:[0][NEDL] }, and @F { Sat:[NEDL][0] }, aim to match
busy weekends, but they omit the `on vacation' shift @F { V }, so
they leave some busy weekends (@F { Sat:DV }, for example) unmatched.
They should be @F { Sat:[NEDL][NEDL] }, @F { Sat:[0V][NEDL] }, and
@F { Sat:[NEDL][0V] }, which are convertible, and in fact they are
equivalent to them because the vacation shift can only be
preassigned to a nurse, not assigned by the solver, and a hand
check shows that cases like @F { Sat:DV } cannot arise.  So the
conversion is hard-wired here.
@PP
Instance @F { HED01.ros } (and also the omitted @F { HED01b.ros })
utilizes @I { conditional constraints }, which require one pattern
to match if another does.  These cannot be converted in general,
but those in these instances can be.  For example, some require
all the shifts taken by a nurse in Week 1 to have the same type,
which is expressible by a cluster busy times constraint with
maximum limit 1 and time groups
@ID lines @Break @F {
"{"1Mon1, 1Tue1, 1Wed1, 1Thu1, 1Fri1, 1Sat1, 1Sun1"}"
"{"1Mon2, 1Tue2, 1Wed2, 1Thu2, 1Fri2, 1Sat2, 1Sun2"}"
"{"1Mon3, 1Tue3, 1Wed3, 1Thu3, 1Fri3, 1Sat3, 1Sun3"}"
}
Again, NRConv is hard-wired to generate suitable
constraints at these file positions.
@PP
The Ikegami instances contain constraints that require sequences
of night shifts to be separated by at least 6 days, expressed by
unwanted patterns penalizing each occurrence of two night shifts
separated by 5, 4, 3, 2, or 1 days (of anything).  NRConv could
express them in the same way, but to reduce the
constraint density it makes them a special case and expresses
them by a single limit active intervals constraint with, for
each day, one negative time group containing the night shift
on that day, and minimum limit 6, with history to ensure that
a sequence at the start of the cycle is not penalized.  This
penalizes the same cases as the unwanted patterns, but with
a different cost in general.  However, good solutions do not
violate these constraints (each has weight 100, which is more
than the total cost of good solutions), so in practice the
amount by which violations are penalized does not matter.
It is true that sequences of, say, 7 consecutive night
shifts are penalized by the original formulation but not
by the converted one, but there are other constraints,
again with weight 100, which limit resources to at most
6 night shifts.
@PP
The Curtois original instances have undocumented features (such as
@F { <Preferred> } limits alongside @F { <Min> } and @F { <Max> },
@F { <CoverWeights> }, and the format of the @F { <CpuTime> }
attribute of solutions) and undocumented interactions (such as
how @F { <DateSpecificCover> } overrides @F { <DayOfWeekCover> }).
Explaining them all is beyond our scope.  There are also documented
features which do not appear in the instances.  For the most part
these are not implemented; NRC will print warning messages and omit
instances that contain them.
@End @Section

@Section
    @Title { The First International Nurse Rostering Competition model }
    @Tag { source.comp1 }
@Begin
@LP
The First International Nurse Rostering Competition
has a simple XML format, documented at
@ID @F "http://www.kuleuven-kortrijk.be/nrpcompetition"
NRConv converts it using models @F { inrc1.xml } and @C { inrc1-soln.xml }.
@PP
Cover constraints appear as numbers of nurses wanted for each shift,
for each skill.  NRC demand constraints are not needed.  There is a
long list of resource constraints.  It all maps easily into NRC,
except as described now.
@PP
@F { <AlternativeSkillCategory> } defines the penalty to apply when
a nurse is assigned to a shift without having the required skill.
This allows each nurse to have a different penalty, which is a
problem for XESTT, since its prefer resources constraint (the
obvious target when converting) associates the penalty with
the shift, giving all unqualified nurses the same penalty.
@PP
This problem is solved as follows.  For each skill @M { s sub i } and each
distinct non-zero weight @M { w sub j } for @F { <AlternativeSkillCategory> },
let @M { S( s sub i, w sub j ) } be the set of all nurses @M { n } such
that the assignment of @M { n } to a place requiring skill @M { s sub i }
should attract penalty @M { w sub j }.  Let @M { N } be the set of all
nurses.  For each place requiring skill @M { s sub i }, define one prefer
resources constraint for each non-zero weight @M { w sub j } such that
@M { S( s sub i, w sub j ) } is non-empty, with weight @M { w sub j }
and set of nurses @M { N - S( s sub i, w sub j ) }.  In practice
this produces just one or two prefer resources constraints per skill.
@PP
There are instance files (for example, @F { long01.xml }), which
assign skills to nurses whose @F { <AlternativeSkillCategory> }
constraint is turned off.  We interpret this to mean that skills
defects are to be ignored for those nurses.
@PP
@F { <CompleteWeekends> } requires a nurse to work on each day of
the weekend, or none.  This is implemented for the first weekend
by a cluster busy times constraint with one time group for each day
of the weekend, containing the times of that day, and minimum limit
equal to the number of days or else 0.  For example, if there are
two days in the weekend, with five times each day, the constraint
has time groups
@ID lines @Break @F {
"{"1Sat1, 1Sat2, 1Sat3, 1Sat4, 1Sat5"}"
"{"1Sun1, 1Sun2, 1Sun3, 1Sun4, 1Sun5"}"
}
and minimum limit 2 or else 0.  This is then repeated for each weekend.
@PP
When weekends have three or more days, it is possible to work on
the first and last days and be free in between them.  The
competition assigns a higher cost for such cases than for other
cases of incomplete weekends.  Although several instances do
have three-day weekends, NRConv does not implement this
refinement.  It can be done using unwanted patterns.
@PP
@F { <IdenticalShiftTypesDuringWeekend> } requires a nurse to
either work the same shift on each day of the weekend, or to be
free on all days.  This is expressed for the first weekend by a
cluster busy times constraint with one time group for each time
of day, containing the weekend's times of that time of day, and
maximum limit 1.  For example, for the two-day weekend with five
times per day, the time groups would be
@ID lines @Break @F {
"{"1Sat1, 1Sun1"}"
"{"1Sat2, 1Sun2"}"
"{"1Sat3, 1Sun3"}"
"{"1Sat4, 1Sun4"}"
"{"1Sat5, 1Sun5"}"
}
This is then repeated for each weekend.  Clearly, if shifts of two
types are busy during one weekend, two time groups will be active
and there will be a violation.
@PP
Although this is logically correct, the competition evaluator does
more:  it treats violations of @F { <CompleteWeekends> } as violations
of this constraint as well.  The cluster busy times constraint just
given does not do this.
@PP
One possible alternative is a limit busy times constraint with
the same time groups as the cluster busy times constraint, but
with a minimum limit of 2 or else 0 applied to each.  This
will penalize the case where (for example) @F { 1Sat1 } is 
busy but @F { 1Sun1 } is not, and vice versa.  The problem here
is that if @F { 1Sat1 } and @C { 2Sun4 } are both busy,
there will be two violations, not one.
@PP
The XESTT @F { Step } cost function could be used to solve this
problem, but unfortunately the intermediate model treats the time
groups of limit busy times constraints as independent of one another,
which they are not when the @F { Step } cost function is used.
@PP
So NRConv generates two constraints for each identical shift types
constraint:  the cluster busy times constraint, plus the equivalent
of a complete weekends constraint.  When there is already a complete
weekends constraint it merges the two, adding their weights together,
provided they have the same hardness and cost function.
@PP
@F { <TwoFreeDaysAfterNightShifts> } requires that on the two days
after a night shift, a nurse should either have the day off or
else work another night shift.  Assuming three shifts per day,
with @F { 3 } being the night shift, our solution makes patterns
@F { [3][12][12]}, @F { [3][12][03]}, and @F { [3][0][12]}
unwanted.  On the second-last day of the cycle, only @F { [3][12]}
is unwanted.
@PP
A violation on both days should cost more than a violation on
one, so @F { [3][12][12]} should get double weight.  However,
the competition evaluator does not do this, so we assign the
same weight to all patterns.  We can then merge the first two,
producing unwanted patterns @F { [3][12]} and @F { [3][0][12]}.
@End @Section

@Section
    @Title { The Second International Nurse Rostering Competition model }
    @Tag { source.comp2 }
@Begin
@LP
The Second International Nurse Rostering Competition, which is documented at
@ID @F "http://mobiz.vives.be/inrc2/"
has a similar format to the first, although with fewer resource
constraints.  NRConv converts it using models @F { inrc2.xml } and
@C { inrc2-soln.xml }.
@PP
The main innovation here is that the competition reflects the way nurse
rosters are often made in reality:  week by week, not all at once.
A @I { weekly instance } is an instance covering one week; a
@I { global instance } covers several weeks.  A global instance is
solved by solving a sequence of weekly instances for consecutive weeks.
Each is hidden from the solver until it has solved the previous weekly
instances.
@PP
XESTT has no representation of a sequence of instances connected by
history.  NRConv produces an XESTT representation of one weekly instance,
based on files giving the general scenario, the week in question,
and history.  It could generate global instances, but one would have
to trust the solve for each week to not look ahead.  History can be
handled by adjusting the limits of the constraints affected, using
one constraint per resource.  NRConv uses XESTT's history features
to generate a single constraint with a history adjustment for each
resource, which is clearer and less verbose.
@PP
One complication with history concerns the order in which the parts
of the source instance are added to the NRC instance.  It can be
convenient to add constraints before nurses, when they precede nurses
in the source model file.  This is done by defining, say, a worker-set
for all the nurses of one contract, passing that worker-set to that
contract's constraints, and adding the nurses to the worker-set later.
But that will not work for constraints affected by history, because
they accept history information for individual nurses.  Adding the
constraints early, then adding their history later would work, but
that is painful to organize.
@PP
Here is the order used by NRConv:
@ID lines @Break @I {
add days (one week's worth), and the one weekend
add shift types
add skills
add contracts
add nurses, including their skills and contracts
add nurse histories to nurses
add worker constraints (shift type, pattern, contract), including history
add cover constraints
add shift-off requests
}
The week file is used only at the end, for cover constraints and
shift-off requests.  But the scenario file is used out of order,
partly to bring everything related to nurses together, but mainly to
ensure that nurses are added before constraints, as explained above;
and the history file is used in the middle of the scenario file.
All this is easy to do because the converter reads all three files
and stores them in memory as @C { KML_ELT } objects before starting
the conversion.  What is not so easy, however, is to recognize the
dependencies and build the NRC instance in a correct order.
@PP
One would think that a Week 0 history file would have zero values
for history, but in fact the supplied Week 0 files have many
non-zero values.  So NRConv fudges and assumes that a Week 0
history file has one week of history.
@End @Section

@Section
    @Title { The Second International Nurse Rostering Competition static model }
    @Tag { source.comp2.static }
@Begin
@LP
Some time after the second international timetabling competition
ended, some papers appeared which tested a particular set of
`static' (multi-week) instances.  So NRConv has been enhanced
to convert these kinds of instances as well.
@End @Section

@Section
    @Title { The Curtois-Qu 2014 model }
    @Tag { source.cq }
@Begin
@LP
Curtois and Qu have recently produced a new set of 24 plain text
instances, documented at
@ID @F "http://www.cs.nott.ac.uk/~psztc/NRP/"
These, and solutions posted by Curtois at the same place, have been
converted using source models @F { cq14.txt } and @F { cq14-soln.txt }.
@PP
Again, much is familiar.  Minimum limits on consecutive busy or
free days do not apply to sequences that include the first or
last day.  This is modelled using XESTT's history mechanism in
a somewhat artificial manner.
@PP
Many resources have a hard limit of 0 on the number of shifts
they can take of a given shift type.  Although this could be
implemented like other workload limits, by a limit workload
constraint, we choose instead to build, for each shift type
@C { st }, the set of all workers with a non-zero workload
limit for shifts of type @C { st }, and generate a prefer
resources constraint for each shift of type @C { st } which
has this set of workers for its preferred set.
@PP
The two approaches are formally equivalent, but the prefer
resources constraints allow solvers to reduce the domains of
shifts to just those workers who have a non-zero workload for
that kind of shift, and so to avoid attempting assignments which
are doomed to fail because of the zero workload limit.  This has
saved running time in the author's tests of his KHE18 solver.
@PP
The solutions to these instances available at the web site
above are not the full set reported in Curtois' paper.
Accordingly we requested and received a larger set from Curtois.
@PP
As in the Curtois original instances, more nurses may be assigned
to a shift than the specified optimum, and NRConv creates extra
event resources to allow for this.  Some
of the solutions received from Curtois, especially for the larger
instances, overload shifts this way to an unreasonable degree.
The worst cases occur in @F { Instance22.Solution.516686.roster}
and @F { Instance22.Solution.516686_1.roster}, which assign 25
nurses to shift @F { d1} on day @F { 140}, when the instance
specifies an optimum cover of 1.
@PP
We have chosen to generate shifts with maximum cover @M { 2c + 5 },
where @M { c } is the optimum cover, making solutions that overload
shifts beyond that point invalid.  Of the 372 solutions received
from Curtois, 42 were rejected for this reason.  The rest were
classified using metadata in the solution files into four solution
groups, one for each algorithm in Table 2 of Curtois' paper.
The best solution for each instance in each solution group was
included in archive @F { CQ14 } (66 solutions altogether).
@End @Section

@EndSections
@End @Chapter
