KHE diary for 2021
==================

At the end of 2019 I was in the middle of implementing the dynamic
programming algorithm for single nurse rostering, aiming to submit
a paper about it to PATAT 2021.

30 December 2020.  Working on the BeginDay, ReceiveValue, and EndDay
  functions.  Done for all expressions except CostSum.  All good.

31 December 2020.  Kept on it.

1 January 2021.  Finished INT_SUM_COMB.  Done boilerplate for
  INT_SEQ and INT_SEQ_COMB.  Added code for building the
  expression trees for limit active intervals monitors.
  Made sure that the children of INT_SEQ and INT_SEQ_COMB
  nodes have increasing last day indexes.  Removed the day
  info type, since only its sig index field is needed now.
  Done INT_SEQ, although it is never used.

2 January 2021.  Finished INT_SEQ_COMB and reinstated the
  code that builds the search tree.  I seem to have a complete
  implementation now, ready to audit and test.  I started work
  on single resource assignment on 4 November 2020, so that is
  two months ago now.  I've struggled some days to get time to
  do any work, which partly explains why it has taken so long.
  Working through a careful audit, have just finished the
  submodules that are the concrete expression types.

3 January 2021.  Auditing the whole thing today.

4 January 2021.  Finished auditing the whole thing today,
  although there are still a few jobs to do.

6 January 2021.  I did work yesterday, on KHE_DAILY_SCHEDULE.
  Today I've revised the documentation, which turned out to
  be in such good shape that I had very little to do.  Now I
  need to finish off KHE_DAILY_SCHEDULE.

7 January 2021.  Finished off KHE_DAILY_SCHEDULE today.

9 January 2021.  Did some reading yesterday, and made some
  small adjustments to the paper.  Did some more of that
  today; I'm pretty happy with the paper as it is now.
  Audited the new task finder code (daily schedules).
  Added code to compare costs when a solution is adopted.

12 January 2021.  Last couple of days went on various home
  duties.  I'm struggling to find decent slabs of code.
  Decided to write some Rolls-Royce debugging code before
  starting to test.

13 January 2021.  Wrote the Rolls-Royce debugging code, have
  clean compile, ready to start testing.

14 January 2021.  Started testing.  Fixed a few little things.

15 January 2021.  OK, now I have a cost discrepancy.  I really
  need some debugging code that works out where that discrepancy
  is coming from.  It needs to re-run the path so it can print
  stuff in detail.

16 January 2021.  Working on KheFullSolnReRun.

17 January 2021.  Implemented history properly now, that is,
  including history after as well as history before.  Ready
  for an audit and test.

18 January 2021.  Auditing history today.  Found one bug, then
  it started to work.  It seems to be working correctly now:
  all 29 best solutions have the true correct cost.

19 January 2021.  Decided to kick off with a graph of running
  time for m days, 0 <= m <= n.  Have got the numbers now, I
  need to convert them into a graph.

20 January 2021.  Working on the first graph.  Decided to
  rewrite the statistics module completely.  For one thing
  it contains external static variables.

21 January 2021.  Rewritten khe_sm_stats.c, and got it working
  again.  Moved KheDateToday to khe_sm_timer.c and consolidated
  all calls on Unix time functions into that file.  Brought the
  documentation up to date with the new interface.  Have finally
  generated a good-looking EPS file.  In fact two, one for running
  time and one for E(m), and both are now in the paper, for a 4
  week instance and an 8 week instance.

24 January 2021.  I've been thinking about full dominance, and
  also writing up the idea of limiting to small total cost as
  a heuristic.  Starting to really work on full dominance today.

25 January 2021.  Added a figure showing an expression tree to the
  paper, and fixed up the full dominance section so that it does
  not gloss over anything.  Basically ready to implement full
  dominance.  I've started in on the boilerplate.  Implemented
  min_assts and max_assts options.

26 January 2021.  Implementing full dominance.

27 January 2021.  Implementing full dominance.  Done the new
  approach to adding dominance tests, which puts an arbitrary
  dom test into each vertex as the vertex is created, leaving it
  to the caller to work out which.  I've also done the working out.

28 January 2021.  Finished implementing full dominance.  I now need
  a careful audit, then I'm ready for testing.

30 January 2021.  Worked over the description of full dominance in
  the PATAT paper (again).  Then did a major reorganization, moving
  signatures to before dominance.

31 January 2021.  Auditing the revised version of the PATAT paper.
  Seems pretty good now, I should go back to strong dominance.

1 February 2021.  Done a first cut of strong dominance, but it
  seems best if I document it carefully in the Guide.  It needs
  to be "set out fair and square, with no contradictions".  Done
  this, now I need to go through and compare the implementation
  with the documentation, and finally study the documentation
  and decide whether or not I agree with it.

2 February 2021.  Verified that the dominance tests in the code
  and the dominance tests in the documentation agree.  Also that
  I think the tests in the documentation are correct now.

3 February 2021.  Revised the testing code, ready to test strong
  dominance.  Done some testing.  I'm getting solutions of the
  same cost as weak dominance, so it seems that the code is
  running well.  However it's turned out that strong dominance
  is much faster than weak dominance!  Who would have thought it?
  And why is it faster, given that it only reduces the size of
  the table by a factor of 3 or 4?

6 February 2021.  Various family things are getting in the way
  at the moment.

9 February 2021.  Can we pare right down so that there is just
  one call for each node on each day it's active?  Why not?
  If the calls are in postorder, the node can get what it
  needs from its children, who will be all done by then.

  Copied my working code to working_khe_sr_single_resource.c,
  and now I am going to try for a one call per active node per
  day version.  Written it up, it's basically ready to implement.

13 February 2021.  Bit of a gap here, owing to family jobs to do
  plus a cold making me low on energy.  Working on boilerplate
  for the new, optimized version of single resource assignment.
  Have clean compile but there is still a lot to add.

15 February 2021.  Written KheSrsExprSetChildIndexes, but it
  needs a careful audit, including a hand trace.  I've also
  written the code that uses child indexes to retrieve the
  child values needed by the parent.

17 February 2021.  Still struggling to get time to work.
  Removed COST_SUM type and replaced root_expr with an
  array of expressions.

  Sorting out debugging, back to a single function with a
  combined interface.  Lots of refinements, all done.

  Audited KheSrsExprSetChildIndexes, including a hand trace.
  It seems to be working, but we'll get debug output too.

18 February 2021.  Auditing the entire new khe_sr_single_resource.c;
  KHE_SRS_FULL_SOLN is next.  It's good: only minor fixes.

19 February 2021.  Completed a full audit of khe_sr_single_resource.c.
  Working on User's Guide.  Just completed a revision of the overview
  of the algorithm section which omits implementation details.

21 February 2021.  Still working on the User's Guide.  Decided to
  cut loose from the PATAT paper, which has its own goals and is
  pretty good as it is.  I may have finished the User's Guide today.
  I need to audit it and think about whether anything is missing,
  or whether the terminology needs tightening up.

22 February 2021.  Audited the User's Guide.  It's pretty good now,
  probably good enough.  Anyway good enough to go on with.

23 February 2021.  Copied running time section from the paper to the
  User's Guide.  Done some testing, am working through a few bugs.
  They may be all fixed now.

24 February 2021.  Bugs all fixed, done some detailed analysis,
  all is good.  HN_0 inevitably incurs a cost of 15 because of
  history problems:  it has to start with time off because it
  has just finished a long run, but then that run ends with only
  2 night shifts and there is a cost of 15 for not having 3.

  Implemented KHE_SRS_EXPR_BUSY_DAY, KHE_SRS_EXPR_FREE_DAY, and
  KHE_SRS_EXPR_WORK_DAY.  All functions defined and added to the
  various case statements, but I am not yet using them when
  converting monitors to expression trees.

25 February 2021.  Not much done today.  But I have added
  BUSY_DAY and FREE_DAY node generation to cluster, active,
  and limit busy times expressions.  Tested it, it seems to
  be working first time, which is great.

26 February 2021.  Added an optimization for cluster busy
  times constraints today.

  Added code to share BUSY_TIME, FREE_TIME, and WORK_TIME
  nodes.  Ready to audit and test.

27 February 2021.  Finished off code to share BUSY_TIME, FREE_TIME,
  WORK_TIME, BUSY_DAY, FREE_DAY, and WORK_DAY nodes.  It's audited,
  tested, and working well.

28 February 2021.  Set up experiment for testing min_assts and
  max_assts.  All done and in the paper now.

  Checked performance of the hash table, and it is not very good.
  I've got debug output which shows that it is examining 14 entries
  during one retrieval, at times.  Surely we can do better than that.

1 March 2021.  Working on hash table.  There seems to be a
  serious problem with it, not sure what yet.  Perhaps the
  table is not being cleared correctly?  I've set up some
  good debug output, not run yet.

2 March 2021.  Working on hash table.  Got the hash function going
  quite well now, and sure enough weak dominance pulled ahead.  But
  then I thought of a way to optimize strong dominance, and weak
  dominance fell behind again.

3 March 2021.  Revised how cost cutoffs are handled, all documented
  and implemented now.  I'm ready to code up the [w, wn, unlimited]
  experiment, by passing a longer *cost_cutoffs array to the test.

4 March 2021.  Ran new code and got the same results, which is good.
  I should do [w, wn, unlimited] test now, but I have had another
  idea which I'll try first.

5 March 2021.  Fixed up KheDataSetWrite.  All done and documented.

  Gearing up for medium dominance.  Now have eq_dom_test_indexes
  array in each day, and a pointer to the enclosing day in each
  partial solution.  So that should be enough to implement with.

6 March 2021.  Added free list of ps sets and finished off medium
  dominance checking.  It's audited and running, but inferior.

7 March 2021.  Implementing trie dominance checking.  I've done
  most of the tricky stuff (KheSrsTrieDominates and so on), I
  just need to do the boilerplate and call the tricky functions.

8 March 2021.  Finished implementing trie dominance checking.
  Tested; after one bug fix it's working well.  Have new graphs,
  they are good.

9 March 2021.  Found a bug with medium dominance testing and
  fixed it.  It's now running slightly faster than strong.

  Sorted monitors so that those with larger limits come first.
  This saves memory in the trie implementation.  And it is faster.

10 March 2021.  Ran the [w, wn, unlimited] test, successfully.
  Worked through the paper (again).  It seems to be finished,
  except that I need to do a few runs on other instances to
  see what that produces.

11 March 2021.  Set up a handy way of testing lots of instances,
  and ran some basic tests.  Got fairly similar results, so I've
  just said that in the paper.  The paper is finished now, although
  I will go through it one more time.

13 March 2021.  Figures inlined, ready for final audit.

14 March 2021.  Final audit of dynamic programming paper completed.

  Removed khe_sr_consec_resource.c from the compilation and commented
  out its documentation in the Guide.  There was a resource
  structural helper module, khe_sr_consec_solver.c, which I have 
  left in place, for the time being anyway.

  Reviewed the User's Guide single resource section; it's good
  enough to go on with.
 
  Off-site backup, also released publicly (Version 2.6).


To Do
=====

  OK, what about this?  Use "extended profile grouping" to group all
  tasks into runs of tasks of the same shift type and domain.  Then
  use resource packing (largest workload resources first) to pack
  the runs into the resources.  Finish off with ejection chains.
  This to replace the current first stage.  Precede profile grouping
  by combinatorial grouping, to get weekend tasks grouped together.  
  Keep a matching at each time, so that unavailable times of other
  resources are taken into account, we want the unassigned tasks at
  every time to be assignable to the unpacked resources at that time.
  At least it's different!

  After INRC2-4-030-1-6291 is done, INRC2-4-035-0-1718 would be good to
  work on.  The current results are 21% worse, giving plenty to get into.

  Event timetables still to do.  Just another kind of dimension?
  But it shows meets, not tasks.

  Ideas:

    * Some kind of lookahead during time sweep that ensures resources
      get the weekends they need?  Perhaps deduce that the max limit
      implies a min limit, and go from there?

    * Swapping runs between three or more resources.  I tried this
      but it seems to take more time than it is worth; it's better
      to give the extra time to ejection chains

    * Ejection beams - K ejection chains being lengthened in
      parallel, if the number of unrepaired defects exceeds K
      we abandon the repair, but while it is less we keep going
      Tried this, it has some interest but does not improve things.

    * Hybridization with simulated annealing:  accept some chains
      that produce worse solutions; gradually reduce the temperature.

  Decided to just pick up where I left off, more or less, and go to
  work on INRC2-4-030-1-6291.  I'm currently solving in just 5.6
  seconds, so it makes a good test.

  Fun facts about INRC2-4-030-1-6291
  ----------------------------------

  * 4 weeks

  * 4 shifts per day:  Early (1), Day (2), Late (3), and Night (4) 
    The number of required ones varies more or less randomly; not
    assigning one has soft cost 30.

  * 30 Nurses:

       4 HeadNurse:  HN_0,  ... , HN_3
      13 Nurse:      NU_4,  ... , NU_16
       8 Caretaker:  CT_17, ... , CT_24
       5 Trainee:    TR_25, ... , TR_29

    A HeadNurse can also work as a Nurse, and a Nurse can also work
    as a Caretaker; but a Caretaker can only work as a Caretaker, and
    a Trainee can only work as a Trainee.  Given that there are no
    limit resources constraints and every task has a hard constraint
    preferring either a HeadNurse, a Nurse, a Caretaker, or a Trainee,
    this makes Trainee assignment an independent problem.

  * 3 contracts: Contract-FullTime (12 nurses), Contract-HalfTime
    (10 nurses), Contract-PartTime (8 nurses).  These determine
    workload limits of various kinds (see below).  There seems
    to be no relationship between them and nurse type.

  * There are unavailable times (soft 10) but they are not onerous

  * Unwanted patterns: [L][ED], [N][EDL], [D][E] (hard), so these
    prohibit all backward rotations.

  * Complete weekends (soft 30)

  * Contract constraints:                   Half   Part   Full    Wt
    ----------------------------------------------------------------
    Number of assignments                   5-11   7-15  15-20*   20
    Max busy weekends                          1      2      2    30
    Consecutive same shift days (Early)      2-5    2-5    2-5    15
    Consecutive same shift days (Day)       2-28   2-28   2-28    15
    Consecutive same shift days (Late)       2-5    2-5    2-5    15
    Consecutive same shift days (Night)      3-5    3-5    3-5    15
    Consecutive free days                    2-5    2-4    2-3    30
    Consecutive busy days                    2-4    3-5    3-5    30
    ----------------------------------------------------------------
    *15-20 is notated 15-22 but more than 20 is impossible.

  Currently giving XUTT a rest for a while.  Here is its to do
  list, prefixed by + characters:

  +Can distinct() be used for distinct times?  Why not?  And also
  +using it for "same location" might work.

  +I've finished university course timetabling, except for MaxBreaks
  +and MaxBlock, which I intend to leave for a while and ponder over
  +(see below).  I've also finished sports scheduling except for SE1
  +"games", which I am waiting on Bulck for but which will not be a
  +problem.

  +MaxBreaks and MaxBlock
  +----------------------

    +These are challenging because they do the sorts of things that
    +pattern matching does (e.g. idle times), but the criterion
    +which determines whether two things are adjacent is different:

      +High school timetabling - adjacent time periods
      +Nurse rostering - adjacent days
      +MaxBreaks and MaxBlock - intervals have gap of at most S.

    +It would be good to have a sequence of blocks to iterate over,
    +just like we have some subsequences to iterate over in high
    +school timetabling and nurse rostering.  Then MaxBreaks would
    +utilize the number of elements in the sequence, and MaxBlock
    +would utilize the duration of each block.

    +We also need to allow for future incorporation of travel time 
    +into MaxBreaks and MaxBlock.  Two events would be adjacent if
    +the amount of time left over after travelling from the first
    +to the second was at most S.

    +Assuming a 15-week semester and penalty 2:

    +MaxBreaks(R, S):

	+<Tree val="sum|15d">
	    +<ForEach v="$day" from="Days">
		+<Tree val="sum:0-(R+1)|2">
		    +<ForEachBlock v="$ms" gap="S" travel="travel()">
			+<AtomicMeetSet e="E" t="$day">
			+<Tree val="1">
		    +</ForEachBlock>
		+</Tree>
	    +</ForEach>
	+</Tree>

    +MaxBlock(M, S):

	+<Tree val="sum|15d">
	    +<ForEach v="$day" from="Days">
		+<Tree val="sum:0-M|2">
		    +<ForEachBlock v="$ms" gap="S" singles="no" travel="travel">
			+<AtomicMeetSet e="E" t="$day">
			+<Tree val="$ms.span:0-M|1s">
		    +</ForEachBlock>
		+</Tree>
	    +</ForEach>
        +</Tree>

    +Actually it might be better if each iteration produced a meet set.
    +We could then ask for span and so forth as usual.  There is also
    +a connection with spacing(a, b).  In fact it would be good to
    +give a general expression which determines whether two
    +chronologically adjacent meets are in the same block.
    +Then we could use "false" to get every meet into a separate
    +block, and then spacing(a, b) would apply to each pair of
    +adjacent blocks in the ordering.  If "block" has the same
    +type as "meet set", we're laughing.

    +I'll let this lie fallow for a while and come back to it.

  +Rather than sorting meets and defining cost functions which
  +are sums, can we iterate over the sorted meets?

  +The ref and expr attributes of time sequences and event sequences
  +do the same thing.

  +There is an example of times with attributes in the section on
  +weighted domain constraints.  Do we want them?  How do they fit
  +with time pattern trees?  Are there weights for compound times?

  +Moved history from Tree to ForEachTimeGroup.  This will be
  +consistent with pattern matching, and more principled, since
  +history in effect extends the range of the iterator.  But
  +what to do about general patterns?  We need to know how each
  +element of the pattern matches through history.

  +Could use tags to identify specific task sets within patterns.

  Install the new version of HSEval on web site, but not until after
  the final PATAT 2020 deadline.

  In the CQ14-13 table, I need to see available workload in minutes.

  Fun facts about instance CQ14-13
  --------------------------------

  * A four-week instance (1Mon to 4Sun) with 18 times per day:

      a1 (1),  a2 (2),  a3 (3),  a4 (4),  a5 (5),
      d1 (6),  d2 (7),  d3 (8),  d4 (9),  d5 (10),
      p1 (11), p2 (12), p3 (13), p4 (14), p5 (15),
      n1 (16), n2 (17), n3 (18)

    There are workloads, presumably in minutes, that vary quite a bit:

      a1 (480),  a2 (480),  a3 (480),  a4 (600),  a5 (720),
      d1 (480),  d2 (480),  d3 (480),  d4 (600),  d5 (720),
      p1 (480),  p2 (480),  p3 (480),  p4 (600),  p5 (720),
      n1 (480),                        n2 (600),  n3 (720)

    480 minutes is an 8-hour shift, 720 minutes is 12 hours.

  * 120 resources, with many hard preferences for certain shifts:

      Preferred-a1 Preferred-a2 Preferred-a3 Preferred-a4 Preferred-a5
      Preferred-d1 Preferred-d2 Preferred-d3 Preferred-d4 Preferred-d5
      Preferred-p1 Preferred-p2 Preferred-p3 Preferred-p4 Preferred-p5
      Preferred-n1 Preferred-n2 Preferred-n3

    although most resources have plenty of choices from this list.
    Anyway this leads to a huge number of prefer resources constraints.

  * There are also many avoid unavailable times constraints, some for
    whole days, many others for individual times; hard and soft.

  * Unwanted patterns (hard).  In these patterns, a stands for
    [a1a2a3a4a5] and so on.

      [d4][a]
      [p5][adp4-5]
      [n1][adp]
      [n2-3][adpn3]
      [d1-3][a1-4]
      [a5d5p1-4][ad]

    This is basically "day off after a sequence of night shifts",
    with some other stuff that probably matters less; a lot of it
    is about the 480 and 720 minute shifts.

  * MaxWeekends (hard) for most resources is 2, for some it is 1 or 3.

  * MaxSameShiftDays (hard) varies a lot, with fewer of the long
    workload shifts allowed.  NB this is not consecutive, this is
    total.  About at most 10 of the shorter, 3 of the longer.
    Doesn't seem very constraining, given that typical workloads
    are 15 or 16 shifts.

  * Many day or shift on requests, soft with varying weights (1-3).

  * Minimum and maximum workload limits in minutes (hard), e.g.

      Minutes           480-minute shifts
      -----------------------------------------------------------
      3120 - 3840
      4440 - 5160
      7440 - 8160        15.5 - 17.0
      7920 - 8640        16.5 - 18.0

    The last two ranges cover the great majority of resources.
    These ranges are quite tight, especially for hard constraints.

  * MinConsecutiveFreeDays 2 (hard) for most resources, 3 (hard)
    for a few.

  * MaxConsecutiveBusyDays 5 (hard) for most resources, 6 (hard)
    for a few.

  * MinConsecutiveBusyDays 2 (hard), for all or most resources.

  Decided to work on CQ14-13 for a while, then tidy up, rerun,
  and submit.

  What does profile grouping do when the minimum limits are
  somewhat different for different resources, and thus spread
  over several constraints?

  INRC1-ML02 would be a good test.  It runs fast and the gap is
  pretty wide at the moment.  Actually I worked on it before (from
  8 November 2019).  It inspired KhePropagateUnavailableTimes.

  Fun facts about INRC1-ML02
  --------------------------

    * 4 weeks 1Fri to 4Thu

    * 4 shifts per day: E (1), L (2), D (3), and N (4).  But there are
      only two D shifts each day, so this is basically a three-shift
      system of Early, Late, and Night shifts.

    * 30 Nurses:
  
        Contract-0  Nurse0  - Nurse7
        Contract-1  Nurse8  - Nurse26
        Contract-2  Nurse27 - Nurse29

    * Many day and shift off requests, all soft 1 but challenging.
      I bet this is where the cost is incurred.

    * Complete weekends (soft 2), no night shift before free
      weekend (soft 1), identical shift types during weekend (soft 1),
      unwanted patterns [L][E], [L][D], [D][N], [N][E], [N][D],
      [D][E][D], all soft 1

    * Contract constraints         Contract-0    Contract-1   Contract-2
      ----------------------------------------------------------------
      Assignments                    10-18        6-14          4-8
      Consecutive busy weekends       2-3     unconstrained     2-3
      Consecutive free days           2-4         3-5           4-6
      Consecutive busy days           3-5         2-4           3-4
      ----------------------------------------------------------------

      Workloads are tight, there are only 6 shifts to spare, or 8 if
      you ignore the overloads in Nurse28 and Nurse29, which both
      GOAL and KHE18x8 have, so presumably they are inevitable.


  Do something about constraints with step cost functions, if only
  so that I can say in the paper that it's done.

  In INRC2-4-030-1-6291, the difference between my 1880 result and
  the LOR17 1695 result is about 200.  About 100 of that is in
  minimum consecutive same shift days defects.  Max working weekends
  defects are another problem, my solution has 3 more of those
  than the LOR17 solution has; at 30 points each that's 90 points.
  If we can improve our results on these defects we will go a long
  way towards closing the gap.

  Grinding down INRC2-4-030-1-6291 from where it is now.  It would
  be good to get a better initial solution from time sweep than I am
  getting now.  Also, there are no same shift days defects in the
  LOR17 solution, whereas there are 

  Perhaps profile grouping could do something unconventional if it
  finds a narrow peak in the profile that really needs to be grouped.

  What about an ejection chain repair, taking the current runs
  as indivisible?

  My chances of being able to do better on INRC2-4-030-1-6291
  seem to be pretty slim.  But I really should pause and make
  a serious attack on it.  After that there is only CQ to go,
  and I have until 30 January.  There's time now and if I don't
  do it now I never will.

  Better to not generate contract (and skill?) resource groups if
  not used.

  Change KHE's general policy so that operations that change
  nothing succeed.  Having them fail composes badly.  The user
  will need to avoid cases that change nothing.

  Are there other modules that could use the task finder?
  Combinatorial grouping for example?  There are no functions
  in khe_task.c that look like task finding, but there are some
  in khe_resource_timetable_monitor.c:

    KheResourceTimetableMonitorTimeAvailable
    KheResourceTimetableMonitorTimeGroupAvailable
    KheResourceTimetableMonitorTaskAvailableInFrame
    KheResourceTimetableMonitorAddProperRootTasks

  KheTaskSetMoveMultiRepair phase variable may be slow, try
  removing it and just doing everything all together.

  Fun facts about COI-Musa
  ------------------------

  * 2 weeks, one shift per day, 11 nurses (skills RN, LPN, NA)

  * RN nurses:  Nurse1, Nurse2, Nurse3,
    LPN nurses: Nurse4, Nurse5, 
    NA nurses:  Nurse6, Nurse7, Nurse8, Nurse9, Nurse10, Nurse11

  Grinding down COI-HED01.  See above, 10 October, for what I've
  done so far.

  It should actually be possible to group four M's together in
  Week 1, and so on, although combinatorial grouping only tries
  up to 3 days so it probably does not realize this.

  Fun facts about COI-HED01
  -------------------------

    * 31 days, 5 shifts per day: 1=M, 2=D, 3=H, 4=A, 5=N

    * Weekend days are different, they use the H shift.  There
      is also something peculiar about 3Tue, it also uses the
      H shift.  It seems to be being treated like a weekend day.
      This point is reflected in other constraints, which treat
      Week 3 as though it had only four days.

    * All demand expressed by limit resources constraints,
      except for the D shift, which has two tasks subject
      to assign resource and prefer resources constraints.
      The other shifts vary between about 7 and 9 tasks.  But
      my new converter avoids all limit resources constraints.

    * There are 16 "OP" nurses and 4 "Temp" nurses.
      Three nurses have extensive sequences of days off.
      There is one skill, "Skill-0", but it contains the
      same nurses as the OP nurses.

    * The constraints are somewhat peculiar, and need attention
      (e.g. how do they affect combinatorial grouping?)
    
        [D][0][not N]  (Constraint:1)
          After a D, we want a day off and then a night shift (OP only).
	  Only one nurse has a D at any one time, so making this should
	  not be very troublesome.

	[not M][D]  (Constraint:2)
	  Prefer M before D (OP only), always seems to get ignored,
	  even in the best solutions.  This is because during the
	  week that D occurs, we can't have a week full of M's.
	  So really this constraint contradicts the others.

	[DHN][MDHAN]  (Constraint:3)
	  Prefer day off after D, H, or N.  Always seems to be
	  satisfied.  Since H occurs only on weekends, plus 3Tue,
	  each resource can work at most one day of the weekend,
	  and if that day is Sunday, the resource cannot work
	  M or A shifts the following week (since that would
	  require working every day).  Sure enough, in the
	  best solution, when an OP nurse works an H shift on
	  a Sunday, the following week contains N shifts and
	  usually a D shift.  And all of the H shifts assigned
	  to Temp nurses are Sunday or 3Tue ones.

	Constraint:4 says that Temp nurses should take H and
	D shifts only.  It would be better expressed by a
	prefer resources constraint but KHE seems happy
	enough with it.

	Constraint:5 says that assigning any shift at all to
	a Temp nurse is to be penalized.  Again, a prefer
	resources constraint would have been better, but at
	present both KHE and the best solution assign 15 shifts
	to Temp nurses, so that's fine.

	The wanted pattern is {M}{A}{ND}{M}{A}{ND}..., where
	{X} means that X only should occur during a week.
	This is for OP nurses only.  It is expressed rather
	crudely:  if 1 M in Week X, then 4 M in Week X.
	This part of it does not apply to N, however; it says
	"if any A in Week X, then at least one N in Week X+1".
	So during N weeks the resource usually has less than
	4 N shifts, and this is its big chance to take a D.

	OP nurses should take at least one M, exactly one D,
	at least one H, at most 2 H, at least one A, at least
	one N.  These constraints are not onerous.

    * Assign resource and prefer resources constraints specify:

        - There is one D shift per day

    * Limit resources constraints specify 

        Weekdays excluding 3Tue

        - Each N shift must have exactly 2 Skill-0 nurses.

	- Each M shift and each A shift must have exactly 4
	  Skill-0 nurses

	- There are no H shifts

	Weekend days, including 3Tue

	- Each H shift must have at least 2 Skill-0 nurses

	- Each H shift must have exactly 4 nurses altogether

	- There are no M, A, or N shifts on 3Tue

	- There are no M, A, or N shifts on weekend days

    * The new converter is expressing all demands with assign
      resource and prefer resources constraints, as follows:

      D shifts:

        <R>NA=s1000:1</R>
	<R>A=s1000:1</R>

	So one resource, any skill.

      H shifts (weekends and 3Tue):

        <R>NA=s1000+NW0=s1000:1</R>
	<R>NA=s1000+NW0=s1000:2</R>
	<R>NA=s1000:1</R>
	<R>NA=s1000:2</R>
	<R>A=s1000:1</R>

	So 2 Skill-0 and 2 arbitrary, as above

      M and A shifts (weekdays not 3Tue):

        <R>NA=s1000+NW0=s1000:1</R>
	<R>NA=s1000+NW0=s1000:2</R>
	<R>NA=s1000+NW0=s1000:3</R>
	<R>NA=s1000+NW0=s1000:4</R>
	<R>W0=s1000:1</R>
	<R>W0=s1000:2</R>
	<R>W0=s1000:3</R>
	<R>W0=s1000:4</R>
	<R>W0=s1000:5</R>

	So exactly 4 Skill-0, no limits on Temp nurses

      N shifts (weekday example)

        <R>NA=s1000+NW0=s1000:1</R>
	<R>NA=s1000+NW0=s1000:2</R>
	<R>W0=s1000:1</R>
	<R>W0=s1000:2</R>
	<R>W0=s1000:3</R>
	<R>W0=s1000:4</R>
	<R>W0=s1000:5</R>

      Exactly 2 Skill-0, no limits on Temp nurses.

  It would be good to have a look at COI-HED01.  It has
  deteriorated and it is fast enough to be a good test.
  Curtois' best is 136 and KHE18x8 is currently at 183.
  A quick look suggests that the main problems are the
  rotations from week to week.

  Back to grinding down CQ14-05.  I've fixed the construction
  problem but with no noticeable effect on solution cost.

  KheClusterBusyTimesConstraintResourceOfTypeCount returns the
  number of resources, not the number of distinct resources.
  This may be a problem in some applications of this function.

  Fun facts about CQ14-05
  -----------------------

    * 28 days, 2 shifts per day (E and L), whose demand is:

           1Mon 1Tue 1Wed 1Thu 1Fri 1Sat 1Sun 2Mon 2Tue 2Wed 2Thu
        ---------------------------------------------------------
        E   5    7    5    6    7    6    6    6    6    6    5
        L   4    4    5    4    3    3    4    4    4    6    4
        ---------------------------------------------------------
        Tot 9   11   10   10   10    9   10   10   10   12    9

      Uncovered demands (assign resources defects) make up the
      bulk of the cost (1500 out of 1543).  Most of this (14 out
      of 15) occurs on the weekends.

    * 16 resources named A, B, ... P.  There is a Preferred-L
      resource group containing {C, D, F, G, H, I, J, M, O, P}.
      The resources in its complement, {A, B, E, K, L, N}, are
      not allowed to take late shifts.

    * Max 2 busy weekends (max 3 for for resources K to P)

    * Unwanted pattern [L][E]

    * Max 14 same-shift days (not consecutive).  Not hard to
      ensure given that resource workload limits are 16 - 18.

    * Many day or shift on requests.  These basically don't
      matter because they have low weight and my current best
      solution has about the same number of them as Curtois'

    * Workload limits (all resources) min 7560, max 8640
      All events (both E and L) have workload 480;
      7560 / 480 = 15.7, 8640 / 480 = 18.0, so every resurce
      needs between 16 and 18 shifts.  The Avail column agrees.

    * Min 2 consecutive free days (min 3 for resources K to P)

    * Max 5 consecutive busy days (max 6 for resources K to P)

    * Curtois' best is 1143.  This represents 2 fewer unassigned
      shifts (costing 100 each) and virtually the same other stuff.

  Try to get CQ14-24 to use less memory and produce better results.
  But start with a smaller, faster CQ14 instance:  CQ14-05, say.

  In Ozk*, there are two skill types (RN and Aid), and each
  nurse has exactly one of those skills.  Can this be used to
  convert the limit resources constraints into assign resource
  and prefer resources constraints?

  Grinding down COI-BCDT-Sep in general.  I more or less lost
  interest when I got cost 184 on the artificial instance, but
  this does include half-cycle repairs.  So more thought needed.
  Could we add half-cycle repairs to the second repair phase
  if the first ended quickly?

  KheCombSolverAddProfileGroupRequirement could be merged with
  KheCombSolverAddTimeGroupRequirement if we add an optional
  domain parameter to KheCombSolverAddTimeGroupRequirement.

  Fun facts about COI-BCDT-Sep
  ----------------------------

    * 4 weeks and 2 days, starting on a Wednesday

    * Shifts: 1 V (vacation), 2 M (morning), 3 A (afternoon), 4 N (night).

    * All cover constraints are limit resources constraints.  But they
      are quite strict and hard.  Could they be replaced by assign
      resource constraints?  (Yes, they have been.)

	  Constraint            Shifts               Limit    Cost
	  --------------------------------------------------------
          DemandConstraint:1A   N                    max 4      10
	  DemandConstraint:2A   all A; weekend M     max 4     100
	  DemandConstraint:3A   weekdays M           max 5     100
	  DemandConstraint:4A   all A, N; weekend M  max 5    hard
	  DemandConstraint:5A   weekdays M           max 6    hard
	  DemandConstraint:6A   all A, N; weekend M  min 3    hard
	  DemandConstraint:7A   all N                min 4      10
	  DemandConstraint:8A   all A; weekend M     min 4     100
	  DemandConstraint:9A   weekday M            min 4    hard
	  DemandConstraint:10A  weekday M            min 5     100
	  --------------------------------------------------------

      Weekday M:   min 4 (hard), min 5 (100), max 5 (100), max 6 (hard),
      Weekend M:   min 3 (hard), min 4 (100), max 4 (100), max 5 (hard) 
      All A:       min 3 (hard), min 4 (100), max 4 (100), max 5 (hard)
      All N:       min 3 (hard), min 4 (10),  max 4 (10),  max 5 (hard)

    * There are day and shift off constraints, not onerous

    * Avoid A followed by M

    * Night shifts are to be assigned in blocks of 3, although
      a four block is allowed to avoid fri N and sat free.  There
      are hard constraints requiring at least 2 and at most 4
      night shifts in a row.

    * At least six days between sequences of N shifts; the
      implementation here could be better, possibly.

    * At least two days off after five consecutive shifts

    * At least two days off after night shift

    * Prefer at least two morning shifts before a vacation period and
      at least one night shift afterwards

    * Between 4 and 8 weekend days

    * At least 10 days off

    * 5-7 A (afternoon) shifts, 5-7 N (night) shifts

    * Days shifts (M and A, taken together) in blocks of exactly 3

    * At most 5 working days in a row.

  Work on COI-BCDT-Sep, try to reduce the running time.  There are
  a lot of constraints, which probably explains the poor result.

  Should we limit domain reduction at the start to hard constraints?
  A long test would be good.

  In khe_se_solvers.c, KheAddInitialTasks and KheAddFinalTasks could
  be extended to return an unassign_r1_ts task set which could then be
  passed on to the double repair.  No great urgency, but it does make
  sense to do this.  But first, let's see whether any instances need it.

  Also thought of a possibility of avoiding repairs during time sweep,
  when the cost blows out too much.  Have to think about it and see if
  it is feasible.

  Take a close look at resource matching.  How good are the
  assignments it is currently producing?  Could it do better?

  Now it is basically the big instances, ERRVH, ERMGH, and MER
  that need attention.  Previously I was working on ERRVH, I
  should go back to that.

  Is lookahead actually working in the way I expect it to?
  Or is there something unexpected going on that is preventing
  it from doing what it has the potential to do?

  UniTime requirements not covered yet:

    Need an efficient way to list available rooms and their
    penalties.  Nominally this is done by task constraints but
    something more concise, which indicates that the domain
    is partitioned, would be better.

    Ditto for the time domain of a meet.

    SameStart distribution constraint.  Place all times
    with the same start time in one time group, have one
    time group for each distinct starting time, and use
    a meet constraint with type count and eval="0-1|...".

    SameTime is a problem because there is not a simple
    partition into disjoint sets of times.  Need some
    kind of builtin function between pairs of times, but
    then it's not clear how this fits in a meet set tree.

    DifferentTime is basically no overlap, again we seem
    to need a binary attribute.

    SameDays and SameWeeks are cluster constraints, the limit
    would have to be extracted from the event with the largest
    number of meets, which is a bit dodgy.

    DifferentDays and DifferentWeeks just a max 1 on each day
    or week.

    Overlap and NotOverlap: need a binary for the amount of
    overlap between two times, and then we can constrain it
    to be at least 1 or at most 0.  NB the distributive law

       overlap(a+b, c+d) = overlap(a, c) + overlap(a, d)
         + overlap(b, c) + overlap(b, d)

    but this nice property is not going to hold for all
    binary attributes.

    Precedence: this is the order events constraint, with
    "For classes that have multiple meetings in a week or
    that are on different weeks, the constraint only cares
    about the first meeting of the class."  No design for
    this yet.

    WorkDay(S): "There should not be more than S time slots
    between the start of the first class and the end of the
    last class on any given day."  This is a kind of avoid
    idle times constraint, applied to events rather than to
    resources (which for us is a detail).
      One task or meet set per day, and then a special function
    (span or something) to give the appropriate measure.  But
    how do you define one day?  By a time group.

    MinGap(G): Any two classes that are taught on the same day
    (they are placed on overlapping days and weeks) must be at
    least G slots apart.  Not sure what to make of this.
    I guess it's overlap(a, b, extension) where extension
    applies to both a and b.

    MaxDays(D): "Given classes cannot spread over more than D days
    of the week".  Just a straight cluster constraint.

    MaxDayLoad(S): "Given classes must be spread over the days
      of the week (and weeks) in a way that there is no more
      than a given number of S time slots on every day."  Just
      a straight limit busy times constraint, measuring durations.
      But not the full duration, rather the duration on one day.

      This is one of several indications that we cannot treat
      a non-atomic time as a unit in all cases.

    MaxBreaks(R,S): "MaxBreaks(R,S) This constraint limits the
      number of breaks during a day between a given set of classes
      (not more than R breaks during a day). For each day of week
      and week, there is a break between classes if there is more
      than S empty time slots in between."  A very interesting
      definition of what it means for two times to be consecutive.

    MaxBlock(M,S): "This constraint limits the length of a block
      of consecutive classes during a day (not more than M slots
      in a block). For each day of week and week, two consecutive
      classes are considered to be in the same block if the gap
      between them is not more than S time slots."  Limit active
      intervals, interpreted using durations rather than times.

  A resource r is busy at some time t if that time overlaps with
  any interval in any meet that r is attending.

  Need a way to define time *groups* to take advantage of symmetries.
  e.g. 1-15{MWF}3 = {1-15M3, 1-15W3, 1-15F3}.  All doubles:
  [Mon-Fr][12 & 23 & 45 & 67 & 78] or something.
  {MWF:<time>} or something.  But what is the whole day anyway?
  All intervals, presumably. {1-15:{MTWRF:1-8}

  See 16 April 2019 for things to do with the XUTT paper.

  It's not clear at the moment how time sweep should handle
  rematching.  If left as is, without lookahead, it might
  well undo all the good work done by lookahead.  But to
  add lookahead might be slow.  Start by turning it off:
  rs_time_sweep_rematch_off=true.  The same problem afflicts
  ejection chain repair during time sweep.  Needs thought.
  Can the lookahead stuff be made part of the solution cost?
  "If r is assigned t, add C to solution cost".  Not easily.
  It is like a temporary prefer resources monitor.

  Here's an idea for a repair:  if a sequence is too short, try
  moving it all to another resource where there is room to make
  it longer.  KheResourceUnderloadAugment will in fact do nothing
  at all in these cases, so we really do need to do something,
  even an ejecting move on that day.

  Working over INRC2-4-030-1-6753 generally, trying to improve
  the ejection chain repairs.  No luck so far.

  Resource swapping is really just resource rematching, only not
  as good.  That is, unless there are limit resources constraints.

  The last few ideas have been too small beer.  Must do better!
  Currently trying to improve KHE18's solutions to INRC2-4-035-2-8875.xml:

    1 = Early, 2 = Day, 3 = Late, 4 = Night
    FullTime: max 2 weekends, 15-22 shifts, consec 2-3 free 3-5 busy
    PartTime: max 2 weekends,  7-15 shifts, consec 2-5 free 3-5 busy
    HalfTime: max 1 weekends,  5-11 shifts, consec 3-5 free 3-5 busy
    All: unwanted [4][123], [3][12], complete weekends, single asst per day
    All: consec same shift days: Early 2-5, Day 2-28, Late 2-5, Night 4-5

    FullTime resources and the number of weekends they work in LOR are:
    
      NU_8 2, NU_9 1, CT_17 1, CT_18 0, CT_20 1, CT_25 1, TR_30 2, TR_32 3

    NB full-time can only work 20 shifts because of max 5 busy then
    min 2 free, e.g. 5-2-5-2-5-2-5-2 with 4*5 = 20 busy shifts.  But
    this as it stands is not viable because you work no weekends.  The
    opposite, 2-5-2-5-2-5-2-5 works 4 weekends which is no good either.
    Ideally you would want 5-2-5-4-5-2-5, which works 2 weekends, but
    the 4 free days are a defect.  More breaks is the only way to
    work 2 weekends, but that means a lower workload again.  This is
    why several of LOR's full-timers are working only 18 hours.  The
    conclusion is that trying to redistribute workload overloads is
    not going to help much.

    Resource types

    HeadNurse (HN_*) can also work as Nurse or Caretaker
    Nurse     (NU_*) can also work as Caretaker
    Caretaker (CT_*) works only as Caretaker
    Trainee   (TR_*) works only as Trainee

  "At least two days off after night shift" - if we recode this,
  we might do better on COI-BCDT-Sep.  But it's surprisingly hard.

  Option es_fresh_visits seems to be inconsistent, it causes
  things to become unvisited when there is an assumption that
  they are visited.  Needs looking into.  Currently commented
  out in khe_sr_combined.c.

  For the future:  time limit storing.  khe_sm_timer.c already
  has code for writing time limits, but not yet for reading.

  Work on time modelling paper for PATAT 2020.  The time model
  is an enabler for any projects I might do around ITC 2019,
  for example modelling student sectioning and implementing
  single student timetabling, so it is important for the future
  and needs to be got right.

  Time sets, time groups, resource sets, and resource groups
  ----------------------------------------------------------

    Thinking about whether I can remove construction of time
    neighbourhoods, by instead offering offset parameters on
    the time set operations (subset, etc.) which do the same.

    Need to use resource sets and time sets a lot more in the
    instance, for the constructed resource and time sets which
    in general have no name.  Maybe replace solution time groups
    and solution resource groups altogether.  But it's not
    trivial, because solution time groups are used by meets,
    and solution resource groups are used by tasks, both for
    handling domains (meet and task bounds).  What about

      typedef struct khe_time_set_rec {
          SSET elems;
      } KHE_TIME_SET;

    with SSET optimized by setting length to -1 to finalize.
    Most of the operations would have to be macros which
    add address-of operators in the style of SSET itself.

       KHE_TIME_SET KheTimeSetNeighbour(KHE_TIME_SET ts, int offset);

    would be doable with no memory allocation and one binary
    search (which could be optional for an internal version).

    I'm leaving this lie for now, something has to be done
    here but I'm not sure what, and there is no great hurry.

  There is a problem with preparing once and solving many times:
  adjustments for limit resources monitors depend on assignments
  in the vicinity, which may vary from one call to another.  The
  solution may well be simply to document the issue.

  At present resource matching is grouping then ungrouping during
  preparation, then grouping again when we start solving.  Can this
  be simplified?  There is a mark in the way.

  Document sset (which should really be khe_sset) and khe_set.

  I'm slightly worried that the comparison function for NRC
  worker constraints might have lost its transitivity now that
  history_after is being compared in some cases but not others.

  Look at the remaining special cases in all.map and see if some
  form of condensing can be applied to them.

  Might be a good idea to review the preserve_existing option in
  resource matching.  I don't exactly understand it at the moment.

  There seem to be several silly things in the current code that are
  about statistics.  I should think about collecting statistics in
  general, and implement something.  But not this time around.

  KheTaskFirstUnFixed is quite widely used, but I am beginning to
  suspect that KheTaskProperRoot is what is really wanted.  I need
  to analyse this and perhaps make some conceptual changes.

  Read the full GOAL paper.  Are there other papers whose aims
  are the same as mine (which GOAL's are not)?  If so I need
  to compare my results with theirs.  The paper is in the 2012
  PATAT proceedings, page 254.  Also it gives this site:

    https://www.kuleuven-kulak.be/nrpcompetition/competitor-ranking

  Can I find the results from the competition winner?  According to
  Santos et al. this was Valouxis et al, but their paper is in EJOR.

  Add code for limit resources monitors to khe_se_secondary.c.

  In KheClusterBusyTimesAugment, no use is being made of the
  allow_zero option at the moment.  Need to do this some time.

  Generalize the handling of the require_zero parameter of
  KheOverloadAugment, by allowing an ejection tree repair
  when the ejector depth is 1.  There is something like
  this already in KheClusterOverloadAugment, so look at
  that before doing anything else.

  There is an "Augment functions" section of the ejection chains
  chapter of the KHE guide that will need an update - do it last.

  (KHE) What about a general audit of how monitors report what
  is defective, with a view to finding a general rule for how
  to do this, and unifying all the monitors under that rule?
  The rule could be to store reported_deviation, renaming it
  to deviation, and to calculate a delta on that and have a
  function which applies the delta.  Have to look through all
  the monitors to see how that is likely to pan out.  But the
  general idea of a delta on the deviation does seem to be
  right, given that we want evaluation to be incremental.

  (KHE) For all monitors, should I include attached and unattached
  in the deviation function, so that attachment and unattachment
  are just like any other update functions?

  Ejection chains idea:  include main loop defect ejection trees
  in the major schedule, so that, at the end when main loop defects
  have resisted all previous attempts to repair them, we can try
  ejection trees on each in turn.  Make one change, produce several
  defects, and try to repair them all.  A good last resort?

  Ejection chains idea:  instead of requiring an ejection chain
  to improve the solution by at least (0, 1), require it to
  improve it by a larger amount, at first.  This will run much
  faster and will avoid trying to fix tiny problems until there
  is nothing better to do.  But have I already tried it?  It
  sounds a lot like es_limit_defects.
