KHE diary for 2018
------------------

1 January 2018.  Working on COI-GPost, recoding it.  I'm
  currently getting 4017; best of 8 is 1285, as is best of 16.

  Saw this in GPost.xml:

    <AppliesToTimeGroup Reference="1Mon1"/>

  It's redundant because the time group has only one
  element.  I've changed NRC to not generate it in this case.

  Done a few more recodings, possibly all that it is feasible
  to do.  Best of 1 is 4023, best of 8 is 1429, as is best of 16.

2 January 2018.  Have a clean compile using the new NRC_LIMIT
  type.  So I should be able to merge constraints now, but I
  need to review the merging algorithm, it might be faulty.

4 January 2018.  Working on the new constraint merging algorithm.
  Have clean compile of the new algorithm, ready for testing.

6 January 2018.  Moved nrmap into nrconv, so that there is only
  one binary to worry about.  Tested the new constraint merging
  algorithm; it seems to be working.

  Had a look through COI-GPost.xml for recoding and merging 
  opportunities.  I could only see one, which would merge
  the minimum and maximum limits on the number of consecutive
  night shifts.  Trouble is, history is involved, which is
  why they are not merging now.

9 January 2018.  Designing NRC_BOUND data type.

11 January 2018.  Made a start on nrc_bound.c, largely cribbed
  from nrc_demand_constraint.c.  Go on like that.

12 January 2018.  Now have clean compile of NRConv using
  NRC_BOUND in demand constraints.  Removed from nrc.h,
  nrc_interns.h, and the documentation.

13 January 2018.  Now have clean compile of NRConv using
  NRC_BOUND in both demand and worker constraints.  Documented
  the algebra of the step constraints.

14 January 2018.  Finished off the algebra for adding step
  constraints when there is both a preferred and a maximum,
  or both a preferred and a minimum.  Not trivial to analyse,
  but it will be simple enough to implement.

15 January 2018.  Revised nrc_bound.c to handle the algebra
  I did yesterday.  All done.

17 January 2018.  Sorted out analysis of allow_zero in
  nrc_bound.c and implemented it.

18 January 2018.  Designed, documented, and implemented
  NrcDemandSetMakeFromBound.

19 January 2018.  Audited NrcDemandSetMakeFromBound, and used
  it in inrc1.c etc.  Type COI_LIMIT, although still needed, has
  been renamed COI_BOUND and simplified by including an NRC_BOUND
  to handle the various cases.  Also did a lot of boilerplate
  changes in coi.c etc. as a consequence.  Have clean compile now.

  Looked at type COI_BOUND, can it be simplified further?
  Who uses the label and var fields?

      CoiBoundVar is used only to check whether there is a
      Var, in which case the limit does not count.  So we
      could handle that by not building a limit in that case.

      CoiBoundLabel is used in three places as a constraint name.

  Looks like we are stuck with COI_BOUND.

20 January 2018.  Audited NrcInstanceConvertWorkerConstraints
  (actually I think I did it before).  All good, I believe.

  I now have what purports to be a complete and correct
  implementation of automatic conversion of demand constraints
  to demands, where feasible.

21 January 2018.  Audited conversion to demands, ready to test.

26 January 2018.  Back at work on XESTT today after taking a few
  days to set up my new web site, which is done now.

  Fixed the problem with Musa.ros:

    <Day>Monday</Day>
      <Cover>
	  <Skill>RN</Skill>
	  <Shift>D</Shift>
	  <Min>1</Min>
	  <Preferred>3</Preferred>
      </Cover>

    <CoverWeights>
        <MinUnderStaffing>7</MinUnderStaffing>
        <MaxOverStaffing>0</MaxOverStaffing>
        <PrefOverStaffing>0</PrefOverStaffing>
        <PrefUnderStaffing>5</PrefUnderStaffing>
      </CoverWeights>

  I've added a note to the NRConv documentation saying that
  it ignores this problem (by ignoring the minimum limit)
  with an error message, and I've implemented that.

27 January 2018.  First solve after generating the new instance,
  cost is 4237, comparable with the 4017 I was getting before.
  Best of 8 is 2217, as is best of 16.  Not great.  But looked
  at in detail, there are now only 4 really bad constraint
  violations, with costs 1000 + 1000 + 100 + 100 = 2200.  So
  things do seem more hopeful.

  Added ClusterBusyTimesConstraintTimeGroupsDisjoint and
  ClusterBusyTimesConstraintTimeGroupsCoverWholeCycle to KHE.
  Used this to get an Avail column into COI-GPost.xml's daily
  timetables.

28 January 2018.  Solving COI, best of 64 runs in a few seconds,
  and its best result has cost 645.  So this is one way to get
  the cost down a fair bit, although not a great way.

29 January 2018.  Working on KHE_TASK_JOB_TYPE today.  All done.
  In KheTaskingAssignResourcesStage1, I removed a call to
  KheTaskingMakeTaskTree, since it seemed to be out of place.

30 January 2018.  Audited the detailed documentation on group
  by frame.  Started to implement, in khe_sr_group_by_frame.c,
  going quite well, basically ready to do the recursive bit.

31 January 2018.  Tested khe_sr_group_by_frame.c as far as it's
  done, all seems to be working.  I now have a list of relevant
  monitors, sorted by start index, with an efficient index into
  this sorted list that will help when building group monitors.
  Also building group monitors now.

1 February 2018.  Now finding the right sets of times, so
  it remains to group the tasks derived from them.

  Constraint merging only merges limit active intervals
  constraints when their history is identical.  We need
  something more nuanced that takes account of the effect
  of history and allows merging when the combined history
  has the same effect.

  I've reread my history paper and established that when
  there is no minimum limit, the value of history_before and
  history_after is irrelevant.  So it is safe to merge two
  constraints with different history_before and history_after
  values, provided one of them has no minimum limit.  The
  merged values must be the values in the constraint with
  the minimum limit.  The merge cannot reduce history_before
  for either constraint, however, since that could invalidate
  some xi.  But who cares?  we only merge when the xi are equal.

  I've implemented this now, and tested it - it's working.

2 February 2018.  Backed up Nonpareil and HSEval today, which
  still leaves the rest of XESTT, notably NRConv, not backed up.
  Should do it real soon.

  Reorganized the group by frame solver so that it is now
  ready to go on to Phase 2, preserving results from Phase 1.
  it can be created just once and go on from Phase 1 to Phase 2.

3 February 2018.  Working on Phase 2 of group by frame.  All
  implemented.  Best of 1 was not very good, but best of 16
  returned a solution with cost 48, which is far and away my
  best result on COI-GPost.xml so far.  But best of 64 is even
  better:  21, in under 5 seconds.  Best of 256 is 17, in 19
  seconds.  This compares quite well with the cost 5 solution
  on Curtois's web site.

  Changed time sweep to look ahead when performing edge adjustment,
  by adding task durations to the number of busy times.  It does
  not seem to make much difference; it might make no difference.

4 February 2018.  PATAT2018 submission deadline is 26 February.
  So I had better start wrapping things up soon.  I don't want
  any accidents at the last minute.

5 February 2018.  Solved the KheTimeUnAssignClashing core
  dumping issue.  As expected, it was a problem using first
  unfixed rather than proper root.  But what is the underlying
  issue here?

  COI-GPost.xml best of 256 now has cost 11, in 20 seconds.  I
  got this by doing the first repair on the grouped tasks.  Best
  of 8 has cost 50, in 0.6 seconds.  Best of 16 has cost 26 in
  1.1 seconds.  These are pretty good results, good enough to
  be going on with.

  Fixed a problem with limit workload constraints, somehow I
  was leaving out the shift set.  Also fixed a stuped problem
  with KheArchiveParallelSolve.

  Did a complete run of all of COI.xml, best of 1 and best of 8.
  Some of the results are quite good (four equal to Curtois'),
  others not so good.  Checked HSEval's evaluations of Curtois'
  solutions, they all agreed, so no major blunders.

  Worked through the history paper, changing the references to
  my web site etc.  It could be submitted as is now.  Ditto
  for the main NRConv paper.

6 February 2018.  Working on the solver paper.  Got it finished
  except for the results.  Audited time sweep to make sure I've
  described it all.
  
  Added "-" as the legal default value of gs_time_limit.
  Changed the name of the ps_time option to ps_time_measure.
  Documented the ps_time_limit option.  Implementing it is next.
  I've done the boilerplate, I'm ready now for the serious stuff.

7 February 2018.  Implemented and tested ps_time_limit, all good.
  Just did a long solve into res_official.xml, looks pretty good.

10 February 2018.  I don't believe there were two days of no work
  here, something has gone wrong with my dates.  Today I did some
  more work on the solve paper, looking up references.  I've emailed
  Tim Curtois about an old paper from South Africa that he cites.

11 February 2018.  Working on an elaborate makefile arrangement
  that will compile the paper.  Fiddled with the various solver
  so that small time limits are better obeyed.  At present it
  takes 102 seconds to solve COI with KHE18 plus KHE18x16, if
  I set the time limits to 1 second.  This looks like it will
  have to do.

  Getting some tables going.  Added integer values to table
  entries, got it all compiling.

12 February 2018.  Finished all the code to allow access to
  the solutions of a solution group by instance, including
  sorting them.  Also finished the changes to HSEval to use
  the new code.  It all should be tested thoroughly but it
  is basically working and that will do for now.

13 February 2018.  Done cost and time in a single table.
  Done a fair amount of general work on the paper.  Changed
  the instance names in CQ14.xml to correct a mistake and
  also make them shorter.  Added KheDeleteSolnGroup(), all
  done and documented.  I need to use it now to omit selected
  solution groups during HSEval.

14 February 2018.  Revised the solve paper.  Implemented averages
  in summary.c, after withdrawing the implementation in table.c.
  Added placeholder tables for INRC1.  Now all seems ready to
  go solving again.

15 February 2018.  Currently have real COI and INRC1 results in
  the paper.  Tried CQ14 but got this:

    ...
    starting first of up to 1 solves of CQ14-20:
    ended last of 1 solves of CQ14-20:
      sole soln of "CQ14-20" has cost 2772.99999
    starting first of up to 1 solves of CQ14-21:
  Fatal error: InsertEdge internal error
  Aborted (core dumped)

16 February 2018.  Yesterday's core dump was due to overflow in
  cost variables in the resource matching, itself due to very
  large hard costs being multiplied by long frame lengths when
  adjusting edges.  Now turning edge adjustment off when this
  is likely to happen.

  Defined and documented KheInstanceAllEventsHavePreassignedTimes
  and KheInstanceMaximumEventDuration, implemented them, and used
  KheInstanceAllEventsHavePreassignedTimes to speed up time
  assignment a lot when all events have preassigned times.

  Defined, documented, and implemented KheEventResourceEquivalent
  as a step towards getting time sweep to run faster on CQ14-21.

  Found and fixed a bug in NRConv for CQ14 which was preventing
  it from generating a limit busy times constraint requiring
  each resource to take at most one shift per day.

17 February 2018.  Tested with equivalenced version of resource
  matching code, and also with one matching per day.  With
  equivalencing code:

    KheGeneralSolve2018 at end (45.24 secs):
    [ Soln (instance "CQ14-21", diversifier 0, cost 9349.99999)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                3443       0.99999
      PreferResourcesMonitor               3754       0.03754
      AvoidUnavailableTimesMonitor           81     642.00002
      ClusterBusyTimesMonitor              1887    2003.00000
      LimitBusyTimesMonitor                  24       0.00076
      LimitActiveIntervalsMonitor           200    6704.00000
      -------------------------------------------------------
      Total                                9389    9349.99999
    ]

  Without equivalencing code:

    KheGeneralSolve2018 at end (45.48 secs):
    [ Soln (instance "CQ14-21", diversifier 0, cost 9344.99999)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                3448       0.99999
      PreferResourcesMonitor               3765       0.03765
      AvoidUnavailableTimesMonitor           79     640.00000
      ClusterBusyTimesMonitor              1898    2007.00000
      LimitBusyTimesMonitor                  27       0.00078
      LimitActiveIntervalsMonitor           200    6697.00000
      -------------------------------------------------------
      Total                                9417    9344.99999
    ]

  It would appear that the equivalencing code has not made
  things faster, but that instating the previously missing
  limit of one shift per resource per day has helped a lot.

  There does seem to have been a small loss of quality
  in adding node equivalence, so I've taken it away again.
  
  Produced a complete paper testing all three data sets.
  It omits CQ14-24 and most of its CQ14 solutions are
  infeasible, but still it's a milestone.

  Optimized setting cutoff times by caching the result of
  the previous call.  Tested, got 36 seconds for CQ14-21,
  which does seem to be an improvement over the previous 46.

  Got this output from time sweep:

     timing:  all = 20.1, cutoffs = 10.1, matches = 10.0

  which is saying that the whole run took 20.1 seconds,
  evenly divided between setting cutoffs and finding matches.
  So both need to be optimized, and nothing else does.

  Started work on a further optimization of cutoff times,
  based on fucntions KheClusterBusyTimesMonitorInitialCutoffTime
  and KheLimitActiveIntervalsMonitorSetCutoffTime.

18 February 2018.  Installed optimized monitor cutoffs into
  time sweep, and got this:

    timing:  all = 11.8, cutoffs = 1.7, matches = 9.0

  which suggests that the running time of cutoffs is not a problem
  now.  The whole run of CQ14-21 took 23.6 secs, which is also an
  improvement over the previous 36 seconds.  Tested on GPost and
  got a good solution  (cost 12), so presumably there are no bugs.

19 February 2018.  Realized that demand nodes contain multiple
  tasks for cases other than preassignment.  This has forced me
  to rethink the design of the revised resource matcher.

  More or less finished khe_mmatch.h and khe_mmatch.c.  The main
  thing left is KheMMatchDemandNodeAssignedTo, or rather the two
  functions that replace it.

20 February 2018.  Audited khe_mmatch.c, all good, Also implemented
  the replacement function for KheMMatchDemandNodeAssignedTo.  In
  the middle of testing it now, currently the edge costs seem wrong.

21 February 2018.  Submitted all three papers today.

23 February 2018.  Fixed the bug, which was a stupid typo in
  khe_mmatch.c in the cost addition function.  Grrr.  Back to
  the expected cost 12 solution for GPost now.

24 February 2018.  Worked on a curious problem with cutoffs,
  documented below, but decided not to do anything about it for
  now.  Added multi-nodes to resource matching.  All implemented
  and audited.  Need to improve KheTaskEquivalent, then test.

25 February 2018.  Elaborated KheTaskEquivalent, all good.
  Tested multi-nodes, found a few bugs, all fixed and working now.
  Started work on refining cutoffs.

26 February 2018.  Sorted out a lot of stuff overnight.  Working
  on documenting the planned implementation in the implementation
  notes section of the KHE guide.  Documentation is done.

27 February 2018.  Implementing the new stuff today.  Going well.

28 February 2018.  Still implementing the new stuff.  All good,
  mostly done now, needs to be compiled/audited.

1 March 2018.  Completed the revised limit active intervals
  monitor, and did a careful audit, which uncovered the fact that
  the history adjustmen was not actually being applied.  I'm
  subtracting it when attaching, and adding it when detaching.

  Wrote a consistency checker, but it is not catching the
  consistency problem, which seems odd.  Even with cutoff_off=true
  there is still a consistency problem, so it's not cutoffs
  that are doing it, apparently.

  Perhaps a final print of all monitor states just before we
  write the solution will turn something up.  The monitor is
  internally consistent but somehow it is not consistent with
  the actual solution.

2 March 2018.  Found an fixed the consistency problem:  it was
  that HSEval was not up to date.  Fixed a few other little things
  and then got this:

    ended last of 8 solves of COI-GPost:
      costs of all solns of "COI-GPost" (8 costs, 7 unique):
	0.00014 0.00018 0.00032 0.00032 0.00035 0.00037 0.00038 0.01051

    ended last of 64 solves of COI-GPost:
      costs of all solns of "COI-GPost" (64 costs, 10 unique):
	0.00014 0.00014 0.00014 0.00014 0.00014 0.00014 0.00014 0.00014
	0.00018 0.00018 0.00018 0.00018 0.00018 0.00018 0.00018 0.00018
	0.00022 0.00022 0.00032 0.00032 0.00032 0.00032 0.00032 0.00032
	0.00032 0.00032 0.00032 0.00032 0.00032 0.00032 0.00032 0.00032
	0.00032 0.00032 0.00035 0.00035 0.00035 0.00035 0.00035 0.00035
	0.00035 0.00035 0.00037 0.00037 0.00037 0.00038 0.00038 0.00038
	0.00038 0.00038 0.00038 0.00038 0.00038 0.00040 0.00042 0.00042
	0.01051 0.01051 0.01051 0.01051 0.01051 0.01051 0.01051 0.01051

  which is not bad although I was hoping for better.  Keep trying.

  Checked that COI-GPost soln with cost 5 is being evaluated correctly.
  Also check the other solutions in COI.xml, all correct except for
  some solutions to CHILD, where the problem is with limit resources
  constraints.  Probably a conversion problem, needs looking into.

  Here is what happens when you change c3 slightly, to represent
  the fraction of the quota used up:

    ended last of 64 solves of COI-GPost:
    costs of all solns of "COI-GPost" (64 costs, 41 unique):
      0.00025 0.00027 0.00028 0.00030 0.00039 0.00042 0.00047 0.00051
      0.00069 0.00214 0.00215 0.00219 0.00220 0.00223 0.00224 0.00225
      0.00226 0.00226 0.00227 0.00228 0.00228 0.00228 0.00229 0.00230
      0.00230 0.00230 0.00230 0.00231 0.00231 0.00234 0.00234 0.00235
      0.00235 0.00235 0.00235 0.00236 0.00237 0.00237 0.00238 0.00238
      0.00238 0.00239 0.00244 0.00245 0.00246 0.00246 0.01013 0.01013
      0.01024 0.01024 0.01024 0.01025 0.01226 0.01226 0.01228 0.01228
      0.01229 0.01229 0.01232 0.01242 0.02013 0.02013 0.02013 0.02031

  It's not nearly as good, not sure why.

3 March 2018.  Fixed the problem with evaluating CHILD that I spotted
  yesterday.  It was a conversion problem, penalizing solutions for
  not covering the 1am - 7.15am interval on the first day, when they
  have no shifts that run during that interval.

  COI-GPost is about as good as it's going to get now.  It might
  be worth spending some time trying to improve the cost 14 soln,
  but it's doubtful whether we can go much lower.  I did get one
  solution of cost 13 by fiddling with the diversity a bit, but
  the general trend was to make things worse.  The best bet is
  to look again at what doit produces with no repairs, and see
  whether that can be improved.  I don't really want to get back
  into ejection chain repairs at the moment.

  Working on bringing cluster busy times monitor up to date.  Have
  clean compile using a slightly revised interface, but the
  implementation needs a careful revision and I'm working on that.

4 March 2018.  Finished off the revisions to the cluster busy
  times constraint today, including a careful audit.  Also made
  some changes to the impl chapter concerning attach and unattach.
  Tested the new cluster busy times monitor implementation.  It
  came out a bit worse on COI-Post, not sure why:

    costs of all solns of "COI-GPost" (8 costs, 6 unique):
      0.00021 0.00022 0.00023 0.00032 0.00032 0.00032 0.00036 0.01016

  Probably worth looking into.

  File khe_sr_group_by_resource.c all written, have a clean compile.
  It needs an audit, then I need to work out where to use it (via an
  option), and then I need to test it.

5 March 2018.  Audited the new grouping by resource code.  Added an
  rs_group_by_resource option for invoking it.  Ready to test.

  Result of doit8 with group by resource:

    costs of all solns of "COI-GPost" (8 costs, 8 unique):
      0.00020 0.00021 0.00022 0.00023 0.00024 0.00028 0.00032 0.00036

  as compared to without it:

    costs of all solns of "COI-GPost" (8 costs, 6 unique):
      0.00021 0.00022 0.00023 0.00032 0.00032 0.00032 0.00036 0.01016

  So there is some improvement here.  It's hard to say whether it is
  owing to grouping by resource, or just because we repair an extra
  time.  However I did a one-off test which repaired twice in all
  cases, with or without grouping by resource the second time, which
  showed that grouping by resource was responsible for the improvement.

  Here are 64 solves with grouping by resource:

    costs of all solns of "COI-GPost" (64 costs, 14 unique):
      0.00014 0.00014 0.00014 0.00016 0.00019 0.00020 0.00020 0.00020
      0.00020 0.00020 0.00021 0.00021 0.00021 0.00021 0.00021 0.00021
      0.00021 0.00021 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00023 0.00023 0.00023 0.00024 0.00024 0.00024
      0.00024 0.00024 0.00024 0.00024 0.00024 0.00028 0.00028 0.00028
      0.00028 0.00028 0.00028 0.00028 0.00028 0.00031 0.00031 0.00032
      0.00032 0.00032 0.00032 0.00032 0.00032 0.00032 0.00032 0.00036
      0.00036 0.00036 0.00036 0.00037 0.01036 0.01036 0.01036 0.01036
    KheArchiveParallelSolve returning (3.4 secs elapsed)

  And here they are without it:

    costs of all solns of "COI-GPost" (64 costs, 13 unique):
      0.00017 0.00019 0.00021 0.00021 0.00021 0.00021 0.00021 0.00021
      0.00021 0.00021 0.00021 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00023 0.00023 0.00023 0.00028 0.00028
      0.00032 0.00032 0.00032 0.00032 0.00032 0.00032 0.00032 0.00032
      0.00032 0.00032 0.00032 0.00032 0.00032 0.00032 0.00032 0.00032
      0.00032 0.00036 0.00036 0.00036 0.00040 0.00042 0.00042 0.00213
      0.00213 0.00213 0.01016 0.01016 0.01016 0.01016 0.01016 0.01016
      0.01016 0.01016 0.01036 0.01036 0.01036 0.01036 0.01036 0.01036
    KheArchiveParallelSolve returning (2.3 secs elapsed)

  There does seem to be a small but worthwhile improvement, although
  with a cost in run time.  It's noticeable that grouping by resource
  gave only 4 solutions with cost over 1000, whereas without it there
  were 14.

  Add functions to cluster busy times and limit active intervals
  monitor that report how many times the monitor has reached a
  maximum limit.  Currently adding these numbers to c3, but it
  might be better to add before and after numbers.

  With at_max_count:

    costs of all solns of "COI-GPost" (8 costs, 8 unique):
      0.00012 0.00015 0.00020 0.00033 0.00215 0.00219 0.00229 0.00233

    costs of all solns of "COI-GPost" (64 costs, 32 unique):
      0.00012 0.00013 0.00015 0.00015 0.00018 0.00018 0.00018 0.00018
      0.00018 0.00020 0.00020 0.00020 0.00020 0.00020 0.00022 0.00022
      0.00022 0.00022 0.00025 0.00026 0.00026 0.00027 0.00033 0.00033
      0.00033 0.00033 0.00034 0.00043 0.00115 0.00212 0.00212 0.00213
      0.00213 0.00215 0.00216 0.00219 0.00219 0.00220 0.00220 0.00223
      0.00223 0.00223 0.00224 0.00224 0.00224 0.00224 0.00225 0.00225
      0.00227 0.00229 0.00229 0.00229 0.00233 0.00233 0.00248 0.00248
      0.00250 0.00253 0.00253 0.00253 0.00417 0.00421 0.01018 0.01114

  Here is before and after at_max.  It's much the same, although the
  worst solutions are a lot better.

    costs of all solns of "COI-GPost" (8 costs, 8 unique):
      0.00013 0.00015 0.00037 0.00213 0.00217 0.00218 0.00222 0.00225

    costs of all solns of "COI-GPost" (64 costs, 32 unique):
      0.00012 0.00013 0.00013 0.00015 0.00015 0.00018 0.00020 0.00020
      0.00022 0.00022 0.00025 0.00025 0.00031 0.00037 0.00037 0.00037
      0.00037 0.00037 0.00037 0.00037 0.00037 0.00039 0.00049 0.00049
      0.00110 0.00110 0.00110 0.00213 0.00214 0.00215 0.00215 0.00217
      0.00217 0.00217 0.00217 0.00218 0.00218 0.00218 0.00218 0.00218
      0.00218 0.00218 0.00218 0.00219 0.00222 0.00225 0.00227 0.00228
      0.00229 0.00230 0.00231 0.00231 0.00231 0.00232 0.00235 0.00236
      0.00236 0.00236 0.00239 0.00243 0.00249 0.00250 0.00250 0.00250
    ] KheArchiveParallelSolve returning (5.1 secs elapsed)

  I'm sticking with before and after at_max.  It's more principled,
  it does not keep fighting old battles.  Here is what happens when
  you give at_max equal weight:

    costs of all solns of "COI-GPost" (64 costs, 28 unique):
      0.00012 0.00013 0.00013 0.00015 0.00018 0.00018 0.00018 0.00018
      0.00018 0.00018 0.00020 0.00022 0.00022 0.00022 0.00025 0.00027
      0.00030 0.00031 0.00037 0.00037 0.00037 0.00037 0.00037 0.00037
      0.00037 0.00037 0.00049 0.00049 0.00110 0.00110 0.00110 0.00213
      0.00213 0.00214 0.00215 0.00215 0.00215 0.00215 0.00217 0.00217
      0.00217 0.00218 0.00218 0.00218 0.00218 0.00218 0.00218 0.00218
      0.00218 0.00219 0.00222 0.00227 0.00228 0.00228 0.00228 0.00228
      0.00232 0.00235 0.00239 0.00243 0.00249 0.00250 0.00250 0.00250
    ] KheArchiveParallelSolve returning (5.3 secs elapsed)

  It's much the same, I won't do this.  This is enough of this stuff,
  I should move on now.

  I've documented a revised KheFrameResourceMaxBusyTimes which can
  find max busy times from limit workload constraints as well as
  from the others it already uses.  So resource matching will not
  have to know where its maximum comes from.  All implemented,
  needs an audit and test.

6 March 2018.  Audited the new KheFrameResourceMaxBusyTimes, it
  seems OK, but something has changed:

    costs of all solns of "COI-GPost" (8 costs, 8 unique):
      0.00013 0.00018 0.00023 0.00024 0.00028 0.00036 0.00037 0.00222

    costs of all solns of "COI-GPost" (64 costs, 20 unique):
      0.00012 0.00013 0.00013 0.00018 0.00018 0.00018 0.00018 0.00018
      0.00020 0.00022 0.00022 0.00022 0.00023 0.00023 0.00023 0.00023
      0.00023 0.00023 0.00023 0.00023 0.00023 0.00023 0.00023 0.00024
      0.00024 0.00024 0.00024 0.00024 0.00024 0.00024 0.00024 0.00025
      0.00027 0.00028 0.00028 0.00028 0.00028 0.00030 0.00031 0.00036
      0.00036 0.00036 0.00036 0.00036 0.00036 0.00036 0.00037 0.00037
      0.00037 0.00037 0.00037 0.00037 0.00037 0.00037 0.00049 0.00049
      0.00110 0.00110 0.00110 0.00214 0.00215 0.00215 0.00222 0.00235

  Anyway some basic tests seem to be working.  Did a few tests on
  COI-BCV-4.13.1 and COI-BCV-3.46.2.  It seems to be working fine
  and giving much the same results as before, only quite a lot faster:

    costs of all solns of "COI-BCV-3.46.2" (8 costs, 5 unique):
      0.00896 0.00896 0.00896 0.00897 0.00897 0.00898 0.00899 0.00900
    ] KheArchiveParallelSolve returning (21.5 secs elapsed)

  This compares with Table 1 in my paper:  895 in 38.9 seconds.
  So it looks like I should resubmit, if only for the faster
  run times.

7 March 2018.  Started testing the CQ14 instancew with CQ14-03.
  Previously KHE18x8 produced an infeasible solution in 6.5
  seconds, but now I am getting this:

    sole soln of "CQ14-03" has cost 0.01408
    ] KheArchiveParallelSolve returning (1.2 secs elapsed)

    costs of all solns of "CQ14-03" (8 costs, 8 unique):
      0.01308 0.01408 0.01409 0.01418 0.01508 0.01609 0.01611 1.01308
    ] KheArchiveParallelSolve returning (2.4 secs elapsed)

    costs of all solns of "CQ14-03" (64 costs, 38 unique):
      0.01208 0.01209 0.01219 0.01308 0.01309 0.01309 0.01316 0.01407
      0.01408 0.01408 0.01408 0.01409 0.01409 0.01411 0.01411 0.01412
      0.01413 0.01413 0.01413 0.01414 0.01415 0.01416 0.01418 0.01506
      0.01507 0.01507 0.01507 0.01508 0.01508 0.01508 0.01509 0.01510
      0.01511 0.01511 0.01512 0.01512 0.01512 0.01512 0.01514 0.01514
      0.01519 0.01605 0.01605 0.01605 0.01607 0.01607 0.01608 0.01608
      0.01608 0.01609 0.01609 0.01609 0.01610 0.01610 0.01611 0.01612
      0.01613 0.01614 1.01306 1.01306 1.01306 1.01308 1.01405 1.01412
    ] KheArchiveParallelSolve returning (19.5 secs elapsed)

  which is in the ballpark for the 0.01001 solutions found by Curtois.

  Tried a complete solve of CQ14.xml (minus CQ14-24) but it was killed
  on CQ14-23 x 8.  So I'm studying CQ14-04, which seems to be the first
  infeasible now:

      sole soln of "CQ14-04" has cost 4.01918
    ] KheArchiveParallelSolve returning (1.6 secs elapsed)

    costs of all solns of "CQ14-04" (8 costs, 8 unique):
      0.02438 2.02328 3.02320 3.02426 3.02443 3.02728 4.01918 4.02128
    ] KheArchiveParallelSolve returning (3.0 secs elapsed)

  Let's investigate this for a while and see if we can improve it.
  CQ14-04 seems like a very nice instance to work on.  It's a four
  week problem with two shifts per day.  There are 10 nurses.  The
  4 infeasibles are 3 cases of three weekends in a row, and one where
  a nurse has been assigned a shift when unavailable.

  The best solutions I have received for CQ14-04 have quite a few
  unassigned shifts, e.g. 4 on 2Sun, 5 on 3Sat, plus 8 on other days,
  making 17 unassigned shifts altogether.  Each costs 100 and explains
  1700 of the 1716 cost of Curtois' best solutions.

  My first solution has 19 unassigned shifts altogether, costing 1900.
  If the 4 hard constraints could be removed, my solution would have
  cost 1918, which is basically two unassigned shifts away from the
  1716 solutions that Curtois has found.  My best of 8 is already 2438.

  Added a list of free task sets to soln, and updated everything
  accordingly.  So now there is no serious memory allocation of
  task sets during solving.

8 March 2018.   Working on KheNurseUnderloadAugment.  The idea is to
  assign several adjacent tasks at once, to get the amount of extra
  workload we need.  I've got something written but it is basically
  a sketch, it needs careful thought.

9 March 2018.  Got KheNurseUnderloadAugment going, although it's a
  bit fragile.  I should really search for defective time groups,
  not retrieve them, because their position in the list could change.
  Anyway I'm now getting this:

    sole soln of "CQ14-04" has cost 2.02431
    ] KheArchiveParallelSolve returning (1.7 secs elapsed)

    costs of all solns of "CQ14-04" (8 costs, 8 unique):
      1.02431 1.02726 1.02750 1.02837 2.02221 2.02327 2.02339 2.02431

  So we are now almost feasible.  The one infeasibility is an
  avoid unavailable times defect, which is basically a fresh
  field to plough.

  Turned out to be very easy to repair avoid unavailable times
  defects:  depending on the model, just call KheNurseOverAugment
  instead of KheOverloadAugment.  The results are:

	sole soln of "CQ14-04" has cost 1.02424
    ] KheArchiveParallelSolve returning (1.5 secs elapsed)

    costs of all solns of "CQ14-04" (8 costs, 8 unique):
      0.02433 0.02533 0.02543 0.02734 0.02740 0.02939 0.03140 1.02424
    ] KheArchiveParallelSolve returning (3.0 secs elapsed)

    costs of all solns of "CQ14-04" (64 costs, 34 unique):
      0.02136 0.02136 0.02136 0.02136 0.02143 0.02344 0.02432 0.02432
      0.02432 0.02432 0.02433 0.02433 0.02433 0.02433 0.02433 0.02433
      0.02433 0.02433 0.02433 0.02433 0.02433 0.02438 0.02438 0.02439
      0.02440 0.02442 0.02450 0.02533 0.02533 0.02533 0.02533 0.02536
      0.02543 0.02543 0.02543 0.02635 0.02636 0.02637 0.02638 0.02639
      0.02734 0.02734 0.02734 0.02734 0.02739 0.02740 0.02740 0.02826
      0.02836 0.02837 0.02839 0.02932 0.02933 0.02939 0.02941 0.03140
      1.02327 1.02327 1.02424 1.02424 1.02533 1.02631 1.02631 1.02631
    ] KheArchiveParallelSolve returning (23.7 secs elapsed)

  So we have reached feasibility in CQ14-04.  My 2433 compares with
  Curtois' 1716, although it looks like 2136 is not hard to reach.
  NB 2433 in practice means 24 assign resource defects and 33
  miscellaneous other defects.

  In principle a solution could be improved by moving a resource
  from one task to another within one meet.  However the planning
  timetable would show these cases as columns where there was an
  assignment in italic font and a non-assignment in italic font,
  and there are none of those in the solution.  So I'll worry
  about this case when it arises.

10 March 2018.  Audited KheNurseTaskAssignOrMoveAugment, it seems
  to be ready to test.

      sole soln of "CQ14-04" has cost 0.02835
    ] KheArchiveParallelSolve returning (1.6 secs elapsed)

    costs of all solns of "CQ14-04" (8 costs, 8 unique):
      0.02432 0.02532 0.02534 0.02540 0.02541 0.02835 1.02329 1.02736
    ] KheArchiveParallelSolve returning (3.2 secs elapsed)

  These results are better, but only marginally.  Allowing ejecting
  moves got us to

      sole soln of "CQ14-04" has cost 0.02231
    ] KheArchiveParallelSolve returning (6.3 secs elapsed)

    costs of all solns of "CQ14-04" (8 costs, 8 unique):
      0.02231 0.02236 0.02238 0.02327 0.02332 0.02530 0.02633 1.02222
    ] KheArchiveParallelSolve returning (11.3 secs elapsed)

  This is the best so far so I suppose I had better stick with it,
  even though it is a lot slower.  After all these assign resources
  constraints are the bottleneck now.

  Encouraging shorter runs gave this:

      sole soln of "CQ14-04" has cost 0.02135
    ] KheArchiveParallelSolve returning (5.0 secs elapsed)

    costs of all solns of "CQ14-04" (8 costs, 8 unique):
      0.02135 0.02228 0.02232 0.02340 0.02526 0.02529 0.02533 0.02546
    ] KheArchiveParallelSolve returning (10.8 secs elapsed)

  It's somewhat faster and somewhat better, so let's stick with it.
  Tried removing at_max_after cost component:

      sole soln of "CQ14-04" has cost 0.02235
    ] KheArchiveParallelSolve returning (5.6 secs elapsed)

    costs of all solns of "CQ14-04" (8 costs, 7 unique):
      0.02136 0.02136 0.02140 0.02235 0.02238 0.02242 0.02337 0.02344
    ] KheArchiveParallelSolve returning (11.0 secs elapsed)

  Slightly slower, slightly worse.  So leave it in.

  Decided to work on CQ14-09 for a while.  Here are first results:

    costs of all solns of "CQ14-09" (8 costs, 8 unique):
      0.04213 1.02911 2.02483 2.03294 3.01893 4.01676 4.02390 4.03093
    ] KheArchiveParallelSolve returning (24.2 secs elapsed)

  This is running with a 20 second time limit, after altering
  the ejection chain solver to make it obey the limit better.
  The best result of 4213 is to be compared with Curtois' result
  of 439.  Quite a long way to go here.  With a 60 second time
  limit the results are quite a lot better:

    costs of all solns of "CQ14-09" (8 costs, 8 unique):
      0.02126 1.01691 1.01914 1.02044 1.02135 2.01004 2.01692 3.02306
    ] KheArchiveParallelSolve returning (66.4 secs elapsed)

  But let's test with the 20 second limit and see what we can
  come up with.

  Found this in Instance9.text:

    D,E=0|D=28|L=28|N=5,8160,7080,5,2,2,2

  But there seems to be nothing equivalent to the N=5 part of it in
  in CQ14-09.xml.  This is saying "at most 5 night shifts", so we
  would expect a limit busy times constraint with time group
  1-4Mon-Sun4 and maximum limit 5.  The explanation is that the
  times have been put in as a group when they should have been
  put in separately.  All fixed now, but what does it do to my
  previous work?

  Here's a redo of CQ14-04:

    sole soln of "CQ14-04" has cost 1.02229
    ] KheArchiveParallelSolve returning (4.9 secs elapsed)

    costs of all solns of "CQ14-04" (8 costs, 8 unique):
      0.02130 0.02340 0.02526 0.02527 0.02533 0.02541 1.02229 1.02327
    ] KheArchiveParallelSolve returning (10.7 secs elapsed)

  So actually not much has changed here.  And here is CQ14-09:

    sole soln of "CQ14-09" has cost 0.02322
    ] KheArchiveParallelSolve returning (20.5 secs elapsed)

    costs of all solns of "CQ14-09" (8 costs, 8 unique):
	0.01584 0.01702 0.02518 0.03009 0.04028 1.01888 1.02214 1.02707
    ] KheArchiveParallelSolve returning (24.0 secs elapsed)

  The best result of 1584 is to be compared with Curtois' result
  of 439.  Quite a long way to go here.  With a 60 second time limit:

    costs of all solns of "CQ14-09" (8 costs, 8 unique):
      0.00898 0.01024 0.01220 0.01612 0.02013 0.02439 1.00786 1.01309
    ] KheArchiveParallelSolve returning (65.4 secs elapsed)

  and this is closer to the ball park.

11 March 2018.  Working on CQ14-23, with a view to getting a half
  decent result within the time limit.  I've put things in place
  to reduce the amount of rematching that is done when the frame
  is longer than one month, and further still when it is longer
  than three months.  KheEjectionChainRepairResources, however,
  I'm just relying on the time limit.

  Did some work on CQ14-23, there are many defects and I am
  a long way from fixing them all.

  I found that I was including all tasks in demand sets, and then,
  because assigning helps with workload, they all get assigned.
  Changed it to include only tasks that actually need assignment.
  Was already doing this with the heuristic solver.

12 March 2018.  Most of the memory cost is in the cluster busy times
  monitors which implement the unwanted shift patterns.  I can halve
  this (for CQ14-23) by grouping shift types that have the same
  unwanted following shift types.

  Indexes:

     10 KHE_MMATCH_EDGE
     11 KHE_MMATCH_NODE
     12 KHE_MMATCH

     20 KHE_EJECTOR
     21 KHE_EJECTOR_MAJOR_SCHEDULE
     22 KHE_EJECTOR_MINOR_SCHEDULE
     23 KHE_SAVED_REPAIR
     24 KHE_AUGMENT
     25 KHE_AUGMENT_INFO
     26 KHE_REPAIR_INFO

     31 KHE_INTERVAL (when copying)
     32 KHE_INTERVAL (KheIntervalAdd)
     33 KHE_LIMIT_ACTIVE_INTERVALS_MONITOR
     
     40 KHE_SUPPLY_NODE (resource matching)
     41 KHE_DEMAND_NODE
     42 KHE_RESOURCE_MATCHING_DEMAND_SET
     43 KHE_RESOURCE_MATCHING_SOLVER

     50 KHE_TASK_SET
     51 KHE_FRAME_MAKE
     52 KheFrameOption thingy
     53 KHE_FRAME_WORKLOAD

     60 KHE_CLUSTER_BUSY_TIMES_MONITOR

13 March 2018.  Today's job is to reduce the memory cost of cluster
  busy times monitors, with a view to getting CQ14-23 to run faster.
  I will also change NRConv so that it generates fewer monitors,
  by grouping together shift types that have the same set of
  prohibited following shift types.

  I'm now making use of spare space in the INHERIT_MONITOR part
  of the monitor, I can squirrel away an extra bool and and extra
  int in there, and this has saved some space in cluster busy times
  monitors, because the allow_zero bool was occupying a whole word.

  Designed, implemented, and documented a new NRC_PATTERN_SET type,
  and used it to reduce the number of patterns.

  Spent quite a lot of time tidying up khe_main.c, including
  adding new options to fiddle with archive files, including
  and excluding instances and solution sets.

  Still crashing when solving CQ14-23 with KHE18x8, although
  it did get right through it (taking 10 minutes despite a
  1 minute time limit) when it was the only instance solved.
  The conclusion is that I really need to implement my own
  arena memory allocator.

14 March 2018.  Working on the new file structure for my
  timetabling code.  Documentation is done, diaries are next,
  then the source code directories and makefiles.

16 March 2018.  Done documentation directories and diaries,
  next comes the source code directories and their makefiles.
  Do it by copying, in case anything goes horribly wrong.
  Got halfway through the solver stuff today.

17 March 2018.  Have clean compile in src_platform, src_solvers,
  and src_khe.

18 March 2018.  Makefiles all done, including in the doc directories
  and in the root directory.  Clean compiles all round.  Added a
  restart target.  Off-site backup of the new KHE done.

  Looked into the anomalous case of KheOptionsFrame, but decided to
  do nothing about it for the time being.  The problem is that it
  has a non-trivial default value.

18 March 2018.  KheArchiveReadFromCommandLine now designed,
  documented, implemented, and used by both khe and hseval.cgi.

  Started work on replacing M by Ha and Hn.  It's going to be
  a long slog.  So far I have flown in the files and replaced
  HaAssert and HaAbort by HnAssert and HnAbort.  I've also
  removed m.h and m.c and am slogging through the consequences.

  I've decided to remove lset and use sset instead, as I have
  already done in many places.

  Converted khe_archive.c to arenas, except there is a problem
  with KML, which does not have an arena-friendly design at
  the moment.

  Documented a revised design for the KML incremental read,
  which needs an audit.

22 March 2018.  In the middle of the new KML incremental read.
  It's on the right track but it's terrible hackwork.

23 March 2018.  Have a clean compile of what seems to be a viable
  version of kml_read.c.  I need to audit it very carefully, but
  it's progress at last.  Some auditing is done.

25 March 2018.  Finished auditing kml_read.c and khe_archive.c;
  they are ready to test.  But first I have to wean everything
  else off the M memory allocator.

26 March 2018.  Working on the conversion from M to Ha.  Basically
  going well; just a few odd things that I should come back to.

28 March 2018.  Still working on the conversion from M to Ha.
  I'm up to khe_monitored_time_group.c and going well,
  although there are a few odd things I need to come back to.

29 March 2018.  Finished converting the platform, except for the
  few problem areas noted below.  Started work on the solvers,
  done a few things, currently up to khe_sm_stats.c.

30 March 2018.  Soldiering on.  I just finished converting the
  layer tree code.  There are too many arena parameters, but
  anyway it's done.  Now up to khe_st_tree_repair.c.

31 March 2018.  Soldiering on.   Now have a clean compile of
  the whole shebang except for

    make[2]: Entering directory `/home/jeff/tt/khe/src_khe'
      make[2]: *** No rule to make target `../src_platform/khe_lset.h',
        needed by `khe_main.o'.  Stop.
    make[2]: Leaving directory `/home/jeff/tt/khe/src_khe'

  Wrote this earlier: "khe_st_meet_bound_group is OK but it would be
  better to move it to "extra types for solving" and have a free list
  in the soln object.  Perhaps rename it meet_bound_set."  But after
  looking at it more closes I've decided to leave it as is.

  Wrote this earlier: "KheKempeMeetMove is OK but it creates an arena
  every time it has to do something non-trivial.  Would it be better
  to pass in an arena as a parameter?"  I looked at this and found
  it used the arena to build a set of meets, so I copied the plan
  of KHE_TASK_SET to produce KHE_MEET_SET and used that instead.
  All implemented, documented, and used in KheKempeMeetMove.

1 April 2018.  Wrote this earlier: "KhePartGraphPartDelete and
  another type get deleted when merging, should there be a better
  way to handle it than to let it leak into the arena?"  I can't
  think of a better way, and anyway they are basically only
  merged so there is no great problem with memory leakage.  In
  the end it all gets freed, when the arena is freed.

  Wrote this earlier: "KhePartitionBinPackAndHow has a memory leak
  right now."  Again, the leak is into the arena.  I've looked
  at how it is used and it is part of building the layer tree;
  the memory will be recycled when the layer tree build ends.

  Wrote this earlier: "KheFrameDelete lost".  Fixed now, via a
  free list of KHE_FRAME_MAKE objects in the enclosing soln.

  Wrote this earlier: "HaArrayDeleteAndPlug may have a problem.
  When we do it we often need to set an index in the entry that
  moved to plug the gap.  But when the removed element happened
  to be at the end, there is no such moved element and no index
  to set.  I really do need to check all calls on this function."
  All checked and a few fixes installed.

  Wrote this earlier: "KheTraceFree still to do."  Actually we
  do not need this function, trace objects get freed, finally,
  with the enclosing solution.  All good now.

  Wrote this before: "KhePathFree still to do.  It looks like we need
  a free list of path objects in the solution and don't have one yet."
  This was based on a misconception:  paths are not created directly
  by the user, and in fact all is fine.

  Wrote this before: "KheEventClassDelete still to do in
  khe_cycle_meet.c.  But it looks as though it might be part of
  a solver, in which case it really should be moved to somewhere
  else altogether and perhaps go into a solver arena."  It does
  seem more like a solver, but everyone needs it presumably, so
  I'm leaving where it is and giving it its own arena.  All good.

  Wrote this before: "KheOrdinaryDemandMonitorDeviationDescription
  puts a very simple string into heap memory.  Does this mean that
  someone is being told that the results of these deviation
  description calls can be freed?"  Yes, they were, but not any
  more, and so I can return predefined strings for these deviation
  descriptions if that suits.  All done.  Heap memory, if required,
  comes from the solution arena.

  Wrote this before: "In khe_event_resource.c, there are two places
  where I have commented out a call to KheResourceGroupDelete.  Need
  to think about what is going on here, and do better."  Fixed now.

2 April 2018.  Wrote this earlier: "KheInstanceFinalize defines a
  scratch arena for use when finalizing, and later deletes it, but
  at present some of the things going into it need to be there
  permanently."  I've now deleted this scratch arena, and also
  removed some arena parameters, so all is well.

  Wrote this before: "Some things, notably time groups, resource
  groups, and event groups, can be made as part of either an
  instance or a solution.  Must make sure that the appropriate
  arena is used in each case."  All done now.

  Wrote this before: "KheFrameWorkloadDelete lost.  The simplest
  thing would be to just give it a free list in soln.c and have
  done with it."  Done.

  Wrote this before: "KheEvennessHandlerDelete is called in other
  places besides KheSolnDelete.  Sorting that out is still to do."
  It's fine.  The evenness handler goes in the solution arena and
  is deleted along with its enclosing solution.  If it is deleted
  earlier by KheEvennessHandlerDelete, it just leaks into the
  arena, which is no big deal.

  Wrote this before: "KheResourceInSolnBeginWorkloadRequirements
  has a problem."  The problem is now fixed, by adding a free
  list of workload requirements to soln.

  Wrote this before: "KheMatchingHallSetFree also used; I have had
  to comment it out for now."  Solved by adding a free list of
  hall set objects to the matching.

3 April 2018.  Started work on new memory arrangements for the
  matching, using KheSolnArena(a).  I have added free lists and
  their corresponding functions to soln, for all matching types,
  and I have used those lists when creating objects of those types,
  and when deleting them.  So that seems to take care of matchings.

5 April 2018.  Still auditing and reorganizing khe_soln.c.

6 April 2018.  Still auditing and reorganizing khe_soln.c.  Have
  all the structural stuff done, which was the main thing, but
  now I plan to go right through the rest.

7 April 2018.  Finished auditing and reorganizing khe_soln.c.  It's
  not brilliant but it is better and it will do for now.  Also fixed
  the documentation of soln group metadata and instance metadata, and
  generally brought the documentation up to date for archives, solution
  groups, and instances.
  
  "KheLimitBusyTimesMonitorSetLowerBound needs revising to take
  account of offsets, apparently." - fixed; in fact, no revision
  was necessary.

  "KheMatchingSupplyChunkMake needs to be augmented to add its own
  supply nodes, apparently." - now done at the call site, not in the
  function.

11 April 2018.  Not sure where the last few days went.  Added

     KheSolnMatchingDeleteWorkloadRequirements(KHE_SOLN soln, KHE_RESOURCE r);

  to balance the workload requirement creation functions; I'm now
  calling it as required.

  Decided that perfect symmetry is not desirable, but rather that
  KheMatchingSupplyChunkDelete should call KheMatchingSupplyNodeDelete.
  Finished converting the matching to free lists and the arena memory.
  This finishes the conversion to arena memory allocation that I
  started on 18 March 2018, except that I have done no testing.
  I said then that it would be a long slog.

  Took an off-site backup today.

16 April 2018.  Several distractions have kept me away from work
  until today.  Working through the User's Guide, checking that
  it agrees with the .h files and tidying it up.  I've just
  finished the "Extra Types for Solving" chapter; "Monitoring"
  is next.

17 April 2018.  Finished working through the User's Guide, except
  not the appendices yet.

19 April 2018.  Looked at Arton Dorneles' bug report today.  Have
  more or less decided that the way forward is to get the current
  version into a publishable state and advise him to upgrade.
  
  Brought the two appendices of the User's Guide up to date,
  including flying in the Howard documentation for Ha and Hn.

20 April 2018.  Converted HSEval.  Designed, documented, and
  implemented some standard functions for converting metadata
  into text, and I'm now using them in HSEval.  Working on
  converting NRC.  Up to nrc_worker_set.c.

21 April 2018.  Finished converting NRC and NRConv.  Made sure
  HaArrayContains is immutable, by copying the implementation
  from M.

22 April 2018.  Reviewed the XESTT web site, then started testing
  and debugging with nrconv.  Found a fair few problems, but I
  have actually succeeded in running NRConv to produce COI.xml,
  INRC1*.xml, and CQ14.xml today, which is great.

23 April 2018.  Testing HSEval today.  Some problems with incremental
  reading.  Hard to say what it is, probably my best bet is to
  recycle my own memory.  Have to enhance HaArena to do that.

25 April 2018.  Working on arena recycling today.  So far I have
  added HaArenaRecycle to howard_a.h, added arena sets to Howard,
  and documented and implemented changes to KHE which allow arenas
  to be recycled while solving.  Changed KheArchiveParallelSolve
  to use this new approach.

  There are scratch arenas used when constructing instances, so
  I need to copy the method used for solutions into instances.

  I also need to investigate reading solutions, and how the
  arenas used then can be recycled.  When I construct a soln
  during the read, it needs to have a shared arena set, so that
  if it is reduced to a placeholder later, the arena freed by
  that can be recycled; at present it is being passed NULL.  This
  arena set could very well be passed on to the parallel solver,
  if it could use it.

26 April 2018.  Still working on arena recycling.  Wrote a nice
  section of the implementation notes appendix to the User's
  Guide about it.  Basically I need one arena set per thread,
  created in the main program even, and I need to pass it down
  the thread as a parameter.

  I've now removed all calls to HaArenaMake and HaArenaDelete
  from the platform directory, and made sure that every call to
  KheArenaSetBegin has a matching call to KheArenaSetEnd.

27 April 2018.  Working on arena recycling in khe/src_solvers.
  All good.  Left a few things behind and started on HSEval.

28 April 2018.  Working on arena recycling.  Finished HSEval.
  Documented parallel solving, have to implement it now.

29 April 2018.  Finished the coding of arena recycling in
  khe_sm_parallel_solve.c.  It needs an audit now.

29 April 2018.  Added code to zero out recycled chunks, which
  I had forgotten before.  Even the chunk containing the arena
  object needs to be zeroed, except not the arena object itself,
  obviously.

  Still working on parallel solving.

2 May 2018.  Finished auditing khe_sm_parallel_solve.c at last.
  Got to work on the arena problem with khe_se_ejector.c and
  decided to pass an arena parameter.  All done and documented.

3 May 2018.  Finished debugging HaArenaRecycle (I hope), and
  successfully ran nrconv on the three sets of instances.  It
  does seem to run faster.  Also tested HSEval, it works, and
  although I have only verified a small sample of the costs so
  far, all those I have checked have been correct.

  Fixed a silly bug in khe_partition.c that was allowing
  removal of the last element of an empty array.

4 May 2018.  Managed to get Arton Doneles' instance, BR-B-23, in
  /home/jeff/tt/arton, working:

    KheGeneralSolve2018 at end (8.35 secs):
    [ Soln (instance "BR-B-23", diversifier 0, cost 40.45179)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignTimeMonitor                       1       1.00000
      DistributeSplitEventsMonitor           44       0.44000
      SpreadEventsMonitor                     3       3.00000
      AvoidClashesMonitor                     5       5.00000
      AvoidUnavailableTimesMonitor           14      31.00000
      LimitIdleTimesMonitor                   7       0.00036
      ClusterBusyTimesMonitor                48       0.01143
      -------------------------------------------------------
      Total                                 122      40.45179
    ]

  Not a great solution, probably, but fast, and better than a crash.
  Running 4 threads to make 4 solutions also worked:

      parallel solve of BR-B-23:  ending last of 4 solves
	costs of all solns of "BR-B-23" (4 costs, 4 unique):
	  25.65200 27.50161 32.55146 40.45179
    ] KheArchiveParallelSolve returning (12.1 secs elapsed)

  Found a problem with recording running times in options, now fixed.
  Found a problem with not passing soln to KheTimeGroupMakeInternal,
  also fixed.

  The good news is that KHE finds feasible solutions for 10 of the
  24 CQ14 instances now.  The bad news is that KH18x8 is terminating
  after the first instance - why?

5 May 2018.  Spent the morning optimizing memory usage in khe_mmatch.c.
  The main target has to be the edge record, since there are more
  edge records than anything else.  I was able to remove two pointers
  from this record, one to the previous edge in the list (I'm using a
  singly linked list now) and the other to the edge's opposite edge
  (I allocate edge pairs in contiguous memory positions now, and
  calculate the offset to the other).  This has reduced its size
  from seven pointers to five pointers, a 30% reduction.  I've done
  some testing, it seems to have worked first time.

  Now making a slightly more efficient use of arenas, but not likely
  to be enough to make a difference.

7 May 2018.  Fixed a bug which arose because I am now dumping core
  when a node has multiplicity zero, and this was occurring when
  there were no demand nodes, as occurs sometimes when rematching.

  Implemented this idea for cycles longer than 28 days: currently,
  time sweep stores all the demand sets, for the whole cycle.  What
  about storing just the current demand set?  We might save a lot
  of memory that way.

8 May 2018.  Changed time indexes in frames to hold only a slice,
  from the first time to the last in the frame.  Should save a lot
  of memory when there are large cycles.

  Successfully ran khe on instances 23 and 24 today.  The solutions
  are not good but at least it's running through to the end well.

  Run whole make of nurse solving paper.  Seems good, have to
  look at the paper next.

9 May 2018.  I'm now solving all instances in all archives, but
  the results have gone bad.  I'll start my attack on that by
  looking at COI-GPost, which went bad along with the rest.

10 May 2018.  Returned the solver to near what it was before,
  by including extra components in the edge function that I
  had deleted previously (at_max and run length).

  I looked at this paper, mentioned by a referee:

    F. Bellanti, G. Carello, F. Della Croce, R. Tadei, A greedy-based
    neighborhood search approach to a nurse rostering problem, European
    Journal of Operational Research, Volume 153, Issue 1, 2004, Pages 28-40

  It is completely irrelevant to my stuff.

11 May 2018.  Found and fixed a problem with audit_and_fix, which was
  causing a type error and subsequent mayhem.  Also fixed the problem
  with the khe main program which was causing audit_and_fix to be called.
  I'm now able to solve XHSTT-2014 using KHE18 and to evaluate the
  resulting archive.

  Revised the modelling and history papers to take account of
  the referees' comments.

12 May 2018.  Today's job:  get Version 2.1 out.  No more excuses.
  I can work on the solver afterwards.

  Assigned a new version number, to apply to all of KHE, with a new
  way of propagating it, which I've documented.

  Checked quickly through the NRConv documentation, there are some
  out of date sections but it is good enough to release.

13 May 2018.  Made a final pass over doc_khe.  Did some other
  tidying up, and released Version 2.1 of KHE.

14 May 2018.  Finalized the modelling and history papers ready
  for camera-ready submission, except that the modelling paper
  makes specific statements about the XESTT web site that I
  need to ensure are true before I resubmit.

15 May 2018.  Revising the solver paper today.  Section 4 (page 7)
  is next.

16 May 2018.  Finished revising the solver paper today.

17 May 2018.  Working on at_max_count today.  Some quick tests on GPost
  suggest that it is well worth having.  I've found and implemented a
  more efficient way of calculating it (going via the resource timetable
  monitor), which should make it efficient enough to use on all instances,
  and I've optimized its calculation in limit active intervals monitors.
  All done and dusted.

  Decided not to do this (it's awkward to design and may not give much):
  Add AtMaxCount to workload limit monitors, only do some heuristic
  thing so that we are at the max when workload gets close to the
  maximum, not necessarily right on the maximum.

19 May 2018.  Worked on this yesterday and today:

    In CQ14, exclude resources from the domains of tasks when they
    have a hard limit of 0 of those kinds of tasks.  This should
    save time when searching for alternative assignments, etc.
    Generate prefer resources constraints instead of limit busy
    times constraints.

    In cq14.c, the call on NrcDemandSetMakeFromBound has a NULL
    preferred_ws.  I need to replace that with the set of all
    resources with a non-zero workload limit for shift type st,
    and change the following penalty to be hard.  Also I need
    to remove the workload limit constraint when the limit is zero.

  All done and seems to be working (I've checked all the evaluations
  in CQ14.xml, they all match with the Curtois ones).  Did the full
  test on CQ14.  The results were better on the whole (just a few
  came out worse), and significantly faster, so that now I am
  getting feasible solutions to all but 6 of the 24 instances,
  instead of all but 8.

  I got GPost back down to 12 by including es_active_augment=swap.
  So I tried it for all of COI, but the results were not great,
  and certainly nowhere near back to the conference submission
  results.  What is going on?

  It looks like my current solution to COI-GPost could be improved
  by moving a pair of day shifts, [4Thu, 4Fri], from H to D, then
  moving [4Sat, 4Sun] from D to H.  Was that tried, and if not why
  not?  Constraint:26/D.  It seems to have tried [4Thu] but not
  [4Thu][4Fri].  In KheActiveIntervalTooShortAugment, I replaced
  deviation by deviation + 1 and got a new best of 8:  cost 11.

20 May 2018.  Added KheTaskAssignmentHasCost and
  KheTaskNonAssignmentHasCost to the platform and used
  KheTaskNonAssignmentHasCost in resource matching.
  There was a bug in the old version that was causing
  redoing to do nothing.

  This redo call:

    calling KheResourceMatchingSolverSolve from KheTimeSweepAssign-
      Resources (b):  1Wed 1Tue 1Mon

  Why doesn't it move the two night shifts to E, F, G, or H?
  Because D has asked for those two night shifts specifically.

  Hurrah!  Found a solution for COI-GPost with cost 10, which
  is my best so far.  Did it by commenting out the calls to
  KheFrameAdjacentToActive in khe_se_solvers.c.  This was
  preventing some useful repairs.  Best of 64 has cost 9,
  as does best of 256.

  Did a full run of COI.xml.  The results have improved but
  quite a lot of them are still worse than the original
  submission; others are better.  More work needed.

  Did a full run of INRC1.  Basically back to the conference
  results, which is good.

  Revised the solver paper.  I may be able to resubmit now.
  Do so if so.

21 May 2018.  I wanted to set up the XESTT web site today, and
  submit the final versions of my conference papers, but I found
  that all my solutions are labelled "KHE Version 2.1", when I
  really need to get things right and label them Version 2.2.
  So I'll tidy everything up, possibly set up the non-KHE part
  of the XESTT web site, and then run the full make again.

  I've decided that the archive files, and the archive names,
  need to include a date.  So must do that before I run make
  again.

  Spent some time tidying up my home version of the XESTT web
  site.  It is more or less ready to go live, given that I
  will be storing my own solutions separately.

22 May 2018.  Finished off the XESTT web site today.  Also did a
  final run through the modelling paper; it's ready to resubmit.

23 May 2018.  It's a big day today, I'm releasing everything:

  - Post final versions of my three PATAT papers on my web site,
    along with the .tex versions as a backup (done)

  - Post my results archive files (done)

  - Post the new XESTT web site (done)

  - Post Version 2.2 of KHE (done).  I've decided not to upgrade
    HSeval on my web site to Version 2.2 just yet, I might have
    more changes to incorporate later.

  - Resubmit the final versions of my three PATAT papers.  Was
    unable to do this because EasyChair is not open for final
    submissions yet; they say Friday

  - Email Gerhard and Greet, saying all done (done)

25 May 2018.  Working on COI-GPost-B.  I compared COI-GPost.ros with
  COI-GPost-B.ros using diff -b, which showed that COI-GPost-B.ros has
  one different weight (109 instead of 100) and no ShiftOn requests.
  So yes, we should be able to do better, not worse.

  Decided to implement the complex constraint merging that will
  replace cluster constraints by limit active intervals constraints
  with minimum limits.  Have a separate module for it, seq_constraint.c.

27 May 2018.  Wrote NrcConstraintConvertActiveToConsecutiveMax today.
  All done, tested, and working well.

28 May 2018.  Submitted final versions of my three PATAT papers today.
  Continued with nrc_condensed.c, I am now successfully building
  pairs of related constraints.  I now have to sort the pairs in
  each bag by increasing penalty, and work out how to generate
  consec constraints that mimic those penalties.

29 May 2018.  Still working on nrc_condensed.c.  All done and
  tested now.  Next step is to try it out on more instances,
  make sure the results are the same, and see which promising
  constraints get left out and whether those can be condensed.

30 May 2018.  Looking at cases that failed to condense.

31 May 2018.  Condensing seems to be working pretty well now.
  I've even been able to remove some special cases from all.map,
  because condensing takes care of them now.  

  Ran the COI archive with the new encodings, and hurrah there was
  a significant overall improvement from condensing (COI-MER
  increased slightly, all the others were either the same or better,
  sometimes a lot better), and no changes in the reported costs of
  the Misc solution set, suggesting that condensing is indeed being
  done exactly.  I've documented it, briefly, in the user's guide.

  Fixed "<Name>M (ax / in ) 7 days off</Name>" and similar cases
  by breaking only at spaces.  All good now.

  Got COI-Millar-2.1 down to cost 0 by refining the repair
  operation for assign resource defects.  It was trying to
  assign tasks that had no cost associated with non-assignment,
  which was never going to be worth doing.  Then did a full
  run of COI.xml, there was a good overall improvement.

1 June 2018.  Sorted out silly names in COI-Azaiez; NRC is now
  making up its own names when the given ones look silly.  All
  done, documented and tested.

  Worked out that COI-Azaiez has [0][123][0] and [123][0][123]
  patterns without the special cases at the start, and so these
  are currently not being condensed.  They need to be condensed
  with non-zero history_before and history_after.  I've implemented
  this and it seems to be working.  Needs a full test now.

2 June 2018.  Sorted out the history problems, I believe.  I've
  checked the Misc solutions and they all have the right costs.
  And I've run the solver on the revised instances, and things
  are improving, although there is a new problem now arising from
  the separation of min limits from max limits due to history.

  Wrote KheGroupTwoMonitors in khe_sr_group_by_frame.c.  It
  needs an audit and test now.

3 June 2018.  Fixed group by frame so that it is able to use
  pairs of constraints, getting a min limit from one and a
  max limit from the other.  Got best of 8 GPost back to cost
  10, which is my usual (pretty good) result.

  Documented what has to be done with constraint merging to
  allow some constraints to have different history after values,
  and implemented it.

  I've found that I'm still not merging the two GPost constraints,
  apparently because one has history_before = 0 and the other has
  history_before = -1.  So I've been thinking about this distinction,
  and I have decided that the default values can be 0, since this
  gives the same effect as having no history at all.  I've checked
  that there are no implementation hurdles, and I've documented it
  in the history paper, and in the KHE Guide (there is nothing to
  change in the NRConv Guide), implemented it all, and found that
  I'm still getting the right results when evaluating COM.xml.
  So that's all good.

4 June 2018.  I've discovered that arena handling in KML is a
  complete and utter mess.  So I am going to have to take some time
  out to sort it all out and get something sensible going.  Sorted
  it out in the platform, now I need to sort it out in NRConv.

5 June 2018.  Finished sorting out the KML arena mess, in the
  platform and in NRConv.  All should be good now.

  Returning to GPost.xml now.  I've done a careful check and
  everything seems to be in order.  I thought I had lost one
  of the labels, but no.

  Compared runtimes for running one thread to get KHE18x1 with
  running four threads.  Pretty much the same (average one second
  per instance longer when running multi-threaded), so I will
  speed up all the runs by running four threads from now on.

6 June 2018.  Working on a scheme for recoding limit busy times
  constraints as cluster busy times constraints, so that they
  can participate in cutoffs.  All designed and documented, I
  just need to implement now.  I've implemented all of it
  except generating the cluster busy times constraints while
  finalizing the limit busy times constraint.

7 June 2018.  Finished limit busy times constraint recoding,
  starting to test.  My first problem is that I don't have
  type-specific constraint debug functions, so I've added
  those.  All done and documented.

8 June 2018.  Audited and tested limit busy times constraint
  recoding.  It seems to have worked first time, and to have
  reduce the KHE18x8 cost of Valouxis-1 from 380 to 180.  I'm
  pretty sure that this is because limit busy times monitors
  with minimum limits are now cluster busy times monitors and
  so they are participating in cutoffs.  All good but need to
  keep working on Valouxis-1, since the COI.xml soln has cost 20.

  Found that when I ask for planning timetables for COI.xml, HSEval
  gives me nothing but a heading.  This turned out to be a
  problem with the order of finalization, which I am in the
  course of fixing by replacing the boolean "complete" with
  an enumerated "finalize" that tracks stages of finalization.

  Valouxis-1 best of 8 is now 180, as stated above.  Best of
  64 is 140, best of 256 is 120.  Hard to see what to do here.

  Got an Avail column into Valouxis, based on an all-negative
  frame with a minimum limit.

  Did a new full run, some things improved, others got worse,
  I think I need to stick with it but go back to the worse
  ones, for example COI-GPost and COI-QMC-1.

  The problem with COI-QMC-1 is just its day or shift on
  requests.  These are obviously not being done first after
  conversion to cluster form.  I've done the documentation,
  and the coding (including taking history into account).
  First test of KHE18x8 reduced cost from about 55000 to 37,
  similar to the 33 I had before.

  Ran the INRC1 tests.  Somewhat lower running time, marginally
  higher costs - but still OK.

  Ran the CQ14 tests.  Fewer feasible solutions than before;
  those that did come out are marginally better.  More to do.

9 June 2018.  I'm making another attempt today to work out
  why CGIPlanningTimetablesHTML2 works on large files like
  COI.xml but CGIPlanningTimetablesHTML does not.

  Replace the cgi.h module with command.h, which is equally happy to
  take its commands from either the command line or from CGI.  This
  should make for a better HSEval program generally, but it will also
  open the way to easier testing of what happens when a CGI script runs.

10 June 2018.  Got the -c command line argument going.  Tried
  various ways to get planning timetables for COI.xml.  When
  I use public_html/cgi-bin/rerun_COI, it works; but when I
  do what should be the exact same thing through the web
  interface, it doesn't work.  So there seems to be nothing
  more that I can do, and I am using the alternative versions
  now that do work.

  Went back to working on instances, tried COI-Ozkarahan.  It
  has limit resources constraints but time sweep was giving
  strange results, i.e. KheResourceMatchingMakeBestAsst.  I've
  rewritten it now and it's doing much better, but as it turns
  out there is one defect of cost 400 in the best solution now,
  so the cost has increased from 100 to 400.  But hopefully
  that will be fixed soon:  it's a weekends thing that needs
  to be recoded (Constraints 1, 2, and 3 in the XESTT version).

  Did a full run to see whether KheResourceMatchingMakeBestAsst as
  revised is helping COI generally.  Average cost is slightly
  worse, but that does not seem to be a good indication of what
  actually happened: 6 got worse, 14 stayed the same, 7 got
  better.  Some of the better ones are a lot better:  Musa
  went from 185 to 175, and 175 is optimal; Ik-3.1.2 went from
  349 to 36.  So it's been good.  More work needed.

11 June 2018.  Recoded the free weekends in COI-Ozkarahan.  I was
  able to re-use an existing specials function, only I had to allow
  it to handle non-consecutive as well as consecutive.  The recoding
  worked first time, and it reduced the cost to 0 first time.  Great.

12 June 2018.  Finished documenting Phase 0 of group by frame, and
  worked on it.  I've got to a point where it is grouping the
  Saturday and Sunday night shifts; that's because Saturday night
  has to be followed by something, and Sunday day is no good
  because that would be a day shift following a night shift.  But
  at present it has not worked out that the Saturday day shift
  should be grouped with the Sunday day shift, presumably because
  it is under the misapprehension that Saturday day can group
  with Sunday night.  So need to look into that.  But in the
  meanwhile I did a full run of COI which was fine.

13 June 2018.  Made Phase 1 of group by frame retry unsuccessful
  times until no more successes.  This meant that it could cycle
  back and try Sat again afer succeeding on Sun.  Big success,
  all 8 solves got cost 0.  So that finishes COI-Millar-2.1.
  A full run is the next thing to try.

  Started work on writing a second section of the impl chapter of
  the NRConv guide, devoted to demand constraints.

14 June 2018.  Working on the second section of the impl chapter
  of the NRConv guide, devoted to demand constraints.

15 June 2018.  Decided that clever conversions from limit resources
  constraints to assign resource and prefer resources constraints
  are not such a great idea.  Instead, I'm now working on a way to
  incorporate limit resources constraints into the bipartite match
  in a way that will often be exact and otherwise can serve as a
  reasonable heuristic.  Documenting this in the time sweep section
  of the User's Guide.

16 June 2018.  Working on a revised specification of resource
  matching which handles limit resources constraints, often
  exactly but at worst heuristically.  Documenting it, it's
  going steadily, I am just getting to the pointy end now.

17 June 2018.  Finished working on a revised specification of resource
  matching which handles limit resources constraints.

18 June 2018.  Now comes the hard part - finding an efficient
  implementation of the revised specification of resource
  matching which handles limit resources constraints.

  I've started the implementation, it is not going to be easy to
  get it efficient.  The plan is to add adjustments to demand
  nodes, and to split demand nodes as their adjustments change
  and force this to happen.  The unadjusted version is safely
  squirreled away in unadjusted_khe_sr_resource_matching.c.

  I've done the boilerplate, the next thing is to do the serious
  stuff, that is, associating adjusments with tasks, efficiently.

19 June 2018.  Did some implementing, then returned to the spec
  and explained carefully the difference between tasks that are
  open to assignment and tasks that are assigned to those tasks,
  and also explained how to handle varying task durations.

  I've more or less decided that each task has to go into its
  own demand node initially.  Demand nodes can be merged at
  the end once their adjustments are known.

20 June 2018.  In the middle of the implementation.  I have
  written up a pretty concrete description of the algorithm,
  and I am in the middle of implementing that.

21 June 2018.  Should be nothing to stop me completing the
  implementation now.  The documentation is very concrete
  and seems to be in great shape.

22 June 2018.  Implementing the very concrete documentation
  today (at last).  Removed the matching_off option, since
  we will always use the matching now that it is handling
  limit resources monitors.  Got the first part of the coding
  done and audited, what remains now is to add adjustments for
  limit resources monitors.

24 June 2018.  Registered for PATAT 2018 today.  Also submitted a
  second revision of the history paper, which is the same as the
  first except that it records the fact that 0 is a sensible
  default value for ai, xi, and ci in all cases.

  Carrying on with limit resources monitors seems to be too hard
  at the moment, so I drafted the overheads for presenting the
  history paper at PATAT today.

25 June 2018.  Continuing with overheads today.  I don't seem to
  have any enthusiasm for real work at the moment.  Updated the
  history paper on the web site.

28 June 2018.  Finished the overheads today, for both the
  modelling paper and the history paper.  I'll look at them
  again close to the time, but they are basically good.  Still
  no enthusiasm for real work, but I'll have to get back to it now.

30 June 2018.  Have been doing some thinking about how assignable
  tasks are distributed into demand nodes.  Need to get this
  bedded down before I go on with the implementation.

  Defined and implemented a KHE_TIME_SET type, and documented it
  in the "extras" chapter.  Used it to replace two existing
  KHE_TIME_SET types, one in rematch, the other in group by frame.

1 July 2018.  Implemented khe_task_grouper.c, which I should be able
  to use to remove grouped nodes (containing tasks which have been
  grouped together because they were initially assigned the same
  resource).  I can just store the leader task of each group now.

2 July 2018.  Integrated khe_task_grouper.c into resource matching,
  and consequently removing grouped nodes.  Made some adjustments to
  the task grouper along the way.

3 July 2018.  Hacking through a revision of khe_sr_resource_matching.c
  to incorporate task profiles today, and generally going on with
  the new version.  All good, and audited what I've done so far.

4 July 2018.  Still revising khe_sr_resource_matching.c.  Actually I
  spent the morning revising the documentation, which is in pretty
  good shape now and follows the intended implementation closely.

5 July 2018.  Still revising khe_sr_resource_matching.c.  Have finally
  made some progress in coding:  I'm now sorting and merging demand
  nodes.

6 July 2018.  Audited what I've done.  It's all good except that I
  have not yet added adjustments representing limit resources
  constraints.  This is of course the hard part.  Fixed the problem
  documented earlier: "There may be a problem with ordering of
  operations, where I visit demand nodes and do the monitor stuff
  before tasks that are assigned the same resource are grouped."

7 July 2018.  Wrote khe_set.c (sset.c without shifts, slices, or
  finalizing).  Using it now in khe_time_set.c instead of sset.c,
  and also in a new type, khe_resource_set.c, which I have made
  in anticipation of needing to intersect resource sets soon.

8 July 2018.  Fleshed out khe_resource_set.c some more, giving
  it more of the functions I will need in khe_resource_matching.c.
  Also used it in khe_resource_group.c, for consistency.  All good.

9 July 2018.  Revised the documentation today to make it match
  the current implementation, and to get ahead a bit.  All good.

10 July 2018.  Still slogging through khe_resource_matching.c.

11 July 2018.  Still slogging through khe_resource_matching.c.
  Did an audit; it's all good as far as it has been done.

13 July 2018.  Actually made some progress today:  I wrote function
  KheDemandNodeDiffCount, which calculates the cardinality of the
  symmetric difference of a resource group with the intersection
  of the resource groups of a task profile, fully optimized as I
  envisaged it needed to be.

14 July 2018.  Now sorting demand nodes by increasing diff_count.
  Diff_count and intersections now handling r0.  When calculating
  diff_count, the incoming resource group always has no r0.  But
  when adding adjustments, there may be an r0.  Added a negative
  domain function to the limit resources constraint.  Removed the
  negative flag.  Will use a negative domain instead.

15 July 2018.  All done except KheDemandNodeSplitToDurn, which will
  be a mess.

17 July 2018.  Done most of KheDemandNodeSplitToDurn.

18 July 2018.  Now allowing NULL in time sets and resource sets, and
  using it in resource matching to simplify the intersect attribute
  of task profiles.

  Off-site backup today.

  I seem to have finished the revised resource matching, although it
  still needs auditing and testing.  Hard to believe it's done:  I
  started on 15 June 2018.  I did do a few other things along the way,
  but it has been a huge job, and I've struggled to motivate myself.

19 July 2018.  Audited the documentation against the code today.

20 July 2018.  Tasks in the same node with different monitored
  durations were being grouped together, which was wrong.  Fixed
  it today, all done and audited, including the documentation.

21 July 2018.  Started testing on GPost, found and fixed a few
  small bugs, then got this:

      parallel solve of COI-GPost:  ending last of 8 solves
	costs of all solns of "COI-GPost" (8 costs, 3 unique):
	  0.00014 0.00022 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026
    ] KheArchiveParallelSolve returning (0.3 secs elapsed)

  So running time is holding up, solution quality has deteriorated.
  Turning KheTaskNonAssignmentHasCost back on gives

    [ "COI-GPost", 8 threads, 8 solves, 6 distinct costs, 0.3 secs:
      0.00010 0.00012 0.00016 0.00018 0.00018 0.00018 0.00036 0.00042
    ]

  which is exactly as before (I've changed the debug print format).

22 July 2018.  Solving more of the COI data set today:

    [ "COI-Ozkarahan", 8 threads, 8 solves, 4 distinct costs, 0.1 secs:
      0.00000 0.00000 0.00100 0.00400 0.00400 0.00400 0.00500 0.00500
    ]

  This is better than the cost 100 solution that KHE18x8 found before.
  Found and fixed a few more small bugs along the way.

  Did a full run of COI.xml, saved it in khe18-07-22.pdf.  It's great,
  nearly every cost is lower than in conf2_khe18.pdf, often a lot
  lower, and the first 5 solutions of KHE18x8 is optimal.  Running
  times are about the same; no problems there.

  I've looked carefully through the list twice.  You could call it
  an unbelievably good result.  I won't do any more testing, I will
  get back on the general grind of looking at solutions and finding
  ways to improve the algorithm.  If that throws up flaws in the
  new resource matcher, I'll fix them then.  Fabulous result.

  Added versions of KheMonitorDebug for specific monitor types.
  They were already written, I just moved them from khe_interns.h
  to khe_platform.h and documented them.

24 July 2018.  Thinking about whether I can remove construction of
  time neighbourhoods, by instead offering offset parameters on
  the time set operations (subset, etc.) which do the same.

    khe_meet.c
      One call to KheTimeGroupNeighbour could actually be replaced
      by a call to KheTimeNeighbour.  The other is part of an
      intersection which will normally contain a single element with
      offset 0.  Nothing insurmountable here, although, in the end,
      there is one case where an offset time group is needed.

    khe_time_group.c
      The sole call to KheTimeGroupNeighbour here occurs in
      KheTimeGroupDomainsAllowAssignment, where it is part of a
      subset test, so moving the offset into that test would work.

    khe_cluster_busy_times_constraint.c
      The call to KheTimeGroupNeighbour here, and in the other
      constraint.c files, could be precompiled.  Only positive
      offsets would be needed, in fact no binary searches would
      be needed since none of the time groups go off the end.

      There aren't going to be many constraints with large sets
      of offsets, so we probably do not need to uniqueify,
      although we could.

  The only use of the instance field in khe_time_set.c is to
  support KHE_TIME KheTimeSetTime(KHE_TIME_SET ts, int i), to
  which we could easily enough add an instance parameter.

25 July 2018.  I've decided to leave time sets and resource sets
  alone for the time being.  Something needs to be done but I
  am not sure what.  Working on COI-WHPP today.  The 4000 cost
  represents four unassigned shifts, which is odd because there
  seems to be plenty of available workload.

  No - day shifts have workload 7, so do evening, but night
  shifts have workload 10.  Available workload is 37 hours,
  unassigned workload is 37 hours, but we have a bin packing
  problem:  the available work is all in small fragments.
  
  All resources have a workload limit of 70 hours; it's a
  soft limit but the cost is 1000, so it will get priority.

  Surely each nurse has to either work 7 night shifts or
  else 10 day/evening shifts?  What other way is there to
  pack workload in exactly?  And indeed this is just what
  the solution from Tim's web site does.

  This instance could probably be solved very well as a
  special case, but do I care?  It might be better to
  just let it go and explain that it is not realistic.

27 July 2018.  Prepared the solver paper overheads today.
  I'll look at them again closer to the time.

28 July 2018.  Wrote some code for requested which should
  do the trick.  Actually it didn't do much at all, so I've
  turned it off for now.

  Prefer resources violation, don't just move to another
  resource, also try assigning to a different task at the
  same time, at least when there are no alternative resources.
  In khe_se_solvers.c, named KheEjectingResourceMoveRepair.
  It seems to have done very little good, but I'll leave it now.

  COI-QMC-1 has the silly "O" shift that means free, I'm
  looking into getting rid of that now.  I've added a
  means_time_off attribute to shift types, called it in
  the right place, and documented that it means the same
  as not assigning a shift at all.  But I'm yet to use
  the new attribute to adjust conversions.

29 July 2018.  Working on removing the"O" shift from COI-QMC-1.
  But I've now decided that it is too hard; the semantics are
  not clear.  It may be that the instance is drawing a distinction
  between a day off and a rostered day off, and we can't lose
  that distinction.

30 July 2018.  Worked to get cluster underload repairs to use
  currently assigned tasks as well as unassigned ones, got this:

    [ "COI-QMC-1", 8 threads, 8 solves, 3 distinct costs, 1.4 secs:
      0.00021 0.00021 0.00021 0.00023 0.00023 0.00025 0.00025 0.00025
    ]

  which is a big improvement on the cost 37 I had before.  So I did
  a full test on the COI instances, and things are somewhat better
  on the whole.  I guess that will do for COI-QMC-1.

1 August 2018.  Auditing the resource repair ejection chain functions.
  Added KheSeResourceSetForEach, which is helpful but not dramatic.
  It's hard to see how to do something similar for task sets, given
  the varying ways they are generated.


  Limit active intervals
  ----------------------
  Depending on the es_active_augment, the initial call is either to
  KheLimitActiveIntervalsMoveAugment or KheLimitActiveIntervalsSwapAugment.
  Both call KheLimitActiveIntervalsAugment, but with a parameter of type
  KHE_ACTIVE_AUGMENT_TYPE to say which kind is wanted.

  For each defective interval, if all positive or all negative, call
  KheActiveIntervalTooLongAugment or KheActiveIntervalTooShortAugment.
  These take parameter aa.  They use the frame functions to strip the
  ends off intervals or add to the ends, calling KheFrameMoveOrSwapRepair
  and KheFrameMoveRepair.  If too short, an attempt is also made to
  move the entire interval to another resource.  It's not quite clear
  why only all positive or all negative intervals are done this way,
  except that it is undoubtedly simpler.  These functions in turn
  call KheFrameMoveRepair and KheFrameSwapRepair, which call
  KheFrameMove either once or twice.  It calls KheTaskSetMove,
  KheKempeTaskSetMoveFrame, and KheEjectingTaskSetMoveFrame,
  which are defined elsewhere.

  Otherwise if too long, a call to KheOverUnderAugment at each position
  is used to try to break it up; if too short, if length is 1 try to
  remove it altogether using KheOverUnderAugment, otherwise try to make
  it longer by extending at each end, again using KheOverUnderAugment.

  Cluster busy times
  ------------------
  The initial call is to KheClusterBusyTimesAugment, which then calls
  either KheClusterUnderloadAugment or KheClusterOverloadAugment.

  KheClusterUnderloadAugment and KheClusterOverloadAugment both call
  KheOverUnderAugment for each inactive/active time group.  When
  repairing times, KheClusterOverloadAugment also has an ejection
  tree repair that is beyond the scope of this review.

  Limit busy times
  ----------------
  The initial call is to KheLimitBusyTimesAugment, which then calls
  KheOverUnderAugment, but for overloaded nurses it then calls
  KheNurseOverAugment instead.

  KheOverUnderAugment, KheOverloadAugment, and KheUnderloadAugment
  ----------------------------------------------------------------
  KheOverUnderAugment calls KheOverloadAugment or KheUnderloadAugment.
  It aims to reduce or increase the amount by which a resource r is
  busy in a time group tg.  KheUnderloadAugment tries to clear out tg
  altogether, if allow_zero.  It then calls KheMoveToResourceAugment,
  first trying tasks that are not currently assigned, then trying tasks
  which are currently assigned.

  KheMoveToResourceAugment calls KheTaskAssignOrMoveAugment, which calls
  KheEjectingTaskAssignOrMoveRepair, which calls KheEjectingTaskMoveFrame.

  KheOverloadAugment calls KheTaskAssignOrMoveAugment and alternatively
  KheEjectingTaskSetUnAssignOrMoveAugment.

  KheNurseOverAugment
  -------------------
  Called by KheAvoidUnavailableTimesAugment and KheLimitBusyTimesAugment.
  It uses the frame iterator to visit sequences of adjacent busy times
  in the current frame, and when they intersect the given time group, it
  tries removing inside left and inside right slices from the frame.
  It's not a good way to do things when the frame is large and the
  time group is small.

  Start again
  -----------
  The basic operation is to move some tasks from whatever they are
  assigned to now to a given resource, or to NULL.  There is also
  a swap of two sets of tasks.

     MoveTasks(tasks, resource)

     SwapTasks(tasks1, tasks2, resource1, resource2) =
       MoveTasks(tasks1, resource1) && MoveTasks(tasks2, resource2)

  The tasks can be defined in various ways:

     * As a single task
     * As a task set
     * As the set of tasks assigned to a given resource r in a time group
     * As the set of tasks assigned to a given resource r in a frame

  What about a type that offers these alternatives, with an iterator?
  Just a set of functions would probably be good enough.

  Here is a framework, let's see how generally it applies:

    TaskMoveAugment(TT1: set of sets of tasks, R2: set of resources)
    {
      for each T1: set of tasks in TT1
      {
	Visit(T1);
	for each r2 in R2
	{
	  Begin repair
	  MoveTasks(T1, r2);
	  End repair
	}
	EndVisit(T1);
      }
    }

    TaskSwapAugment(TT1: set of sets of tasks, R2: set of resources)
    {
      for each T1: set of tasks in TT1
        if( !Visited(T1) )
	{
	  Visit(T1);
	  r1 = the resource (or NULL) that all tasks of T1 are assigned
	  for each r2 in R2
	  {
	    for each T2, where T2 is assigned r2 and matches with T1
	    {
	      Begin repair
	      MoveTasks(T1, r2) && MoveTasks(T2, r1);
	      End repair
	    }
	  }
	  EndVisit(T1);
	}
    }

  TaskMoveAugment includes assignment, when the tasks of T1 are
  unassigned.  It also includes unassignment, when R2 contains NULL.
  And it includes Kempe moves, by means of a MoveTasks(T1, r2) that
  is more complex than a simple move, and indeed involves identifying
  sets of clashing tasks S2 and swapping them, so that, as for swapping,
  each task of T1 must be assigned the same resource r1.

  When swapping, each set of tasks T1 in TT1 must be assigned the same
  resource r1, or NULL.  And there has to be a way to generate sets of
  alternative task sets T2 which are all assigned r2 and which match in
  some way with T1 (same times, same total workload, whatever).

  These frameworks just give the general structure; they omit details
  such as how TT1 is represented and how it is iterated over, whether
  R2 contains NULL (meaning that unassignment is allowed), and so on.

  NurseOverAugment - TT1 has one task set for each active interval of
  r's timetable which overlaps tg.  When visiting T1, we try repairs on
  T1's prefixes and suffixes, not just on T1 itself.  R2 is the set of
  all resources minus r and plus NULL.

  KheActiveIntervalTooLongAugment - TT1 is arguably the set of all
  overlong intervals, but by the time KheActiveIntervalTooLongAugment
  is called we are into the first loop and have a single T1.  Again
  we visit prefixes and suffixes of this, and again R2 is the set
  of all resources minus r and plus NULL.

  KheActiveIntervalTooShortAugment - much the same, but this time
  the prefixes and suffixes are just outside T1, not inside it.

  KheFrameMoveRepair - this is essentially

    Begin repair
    MoveTasks(T1, r2);
    End repair

  with the tasks identified by a frame plus a resource (everything
  in the frame assigned the resource), and a clever swap of the
  from and to resources to handle negative frames.  KheFrameMove
  is MoveTasks(S, r) with various options.

  KheOverloadAugment - this is TaskMoveAugment with TT1 being
  the set of sets of singleton tasks assigned to r during tg,
  and R2 being all resources.  It calls KheTaskAssignOrMoveAugment,
  which is essentially

    Visit(T1);
    for each r2 in R2
    {
      Begin repair
      MoveTasks(T1, r2);
      End repair
    }
    EndVisit(T1);

  for T1 being one of these tasks.  When zero is required and
  there is more than one tasks it uses a task set move, calling
  KheEjectingTaskSetUnAssignOrMoveAugment with essentially the
  same structure as KheTaskAssignOrMoveAugment.

  KheUnderloadAugment - this calls two other augment functions,
  KheOverloadAugment(when allow_zero), and two calls to
  KheMoveToResourceAugment.  Here TT1 is the set of singleton
  sets each containing one task running at a time of tg, and
  R2 is just the one resource we want to move to.

  Kinds of move:

    (r    --> NULL)  Unassignment only
    (NULL --> r)     Assignment and ejecting assignment
    (r1   --> r2)    Move, ejecting move, Kempe move, and swap

  Look at this from KheFrameMove:

    res = (to_r == NULL ? KheTaskSetMove(ts, NULL) :
      kempe_move ? KheKempeTaskSetMoveFrame(ts, to_r, days_frame) :
      KheEjectingTaskSetMoveFrame(ts, to_r, allow_eject, days_frame));

  it's pretty clear that a cleanup is needed.  And there are lots
  of kemp_move and allow_eject parameters.  Ultimately, allow_eject
  ends up in function KheTimeUnAssignClashing, which unassigns
  clashing tasks, but only if allow_eject is true.  Clearly we
  need a task move and task set move operations with a parameter
  giving the type of move:  basic (fail if clashes), ejecting
  (unassign if clashes), Kempe.

2 August 2018.  Added KHE_MOVE_TYPE and used it to reduce and
  simplify the parameters in the task repair operations of the
  ejection chain solver.  But swapping is still separate, and
  for good reason (see above).
  
3 August 2018.  Working on levels in the ejection chain code today.

    TaskMoveAugment(TT1: set of sets of tasks, R2: set of resources)
    {
      for each T1: set of tasks in TT1        /* 1 */
	if( !Visited(T1) )                    /* 2 */
	{
	  Visit(T1);
	  for each r2 in R2                   /* 3 */
	  {
	    Begin repair                      /* 4 */
	    MoveTasks(T1, r2);                /* 5 */
	    End repair
	  }
	  EndVisit(T1);
	}
    }

  Level 1:  visit and change a set of zero or more variables
  Level 2:  visit and change one variable
  Level 3:  change one variable by applying a set of alternative repairs
  Level 4:  apply one repair to one variable or set of variables
  Level 5:  change a variable or set of variables

  Implemented and documented the fact that all typed task moves do an
  unassignment when r is NULL.  Did some more fiddling with the ejection
  chain solver to bed down typed task moves.  All good.

4 August 2018.  Added KheTypedMeetMove and KheTypedMeetMoveTime.  All
  done and documented.  However I was somewhat stymied when I tried
  to use them in khe_se_solvers.c, because of the way that different
  kinds of moves are tried one after the other by KheMeetAugment.  I'm
  doing it but the simplification is not as radical as I had hoped.

  Finally sorted out a plan for levels, which I've documented at the
  top of khe_se_solvers.c.  It has 5 levels:

  Level 1:  Full augment functions (*Augment)
  Level 2:  Partial augment functions (*Augment)
  Level 3:  Multi-repair functions (*MultiRepair)
  Level 4:  Repair functions (*Repair)
  Level 5:  Solution changing functions, usually defined elsewhere.

  Moved task set visiting functions to khe_task_set.c, and added
  corresponding meet set visiting functions to khe_meet_set.c.

5 August 2018.  Moved a lot of the frame solver code to a separate file,
  khe_sr_frame.c.  All done and documented.  Done quite a lot of tidying
  up in khe_se_solvers.c, including reorganizing the functions into one
  submodule for each class of repairs, containing repair, multi-repair,
  and augment functions.  It's compiling but needs an audit.

  Had a good idea for a general approach to nurse rostering repairs:
  give them the same general structure as the others, but at the
  last minute, when we are down to moving one task from r1 to r2,
  get tasks currently assigned to r1 in adjacent time groups of
  the current frame, and move a whole set of tasks.  I've documented
  this idea in the User's Guide, here is a guide to how closely my
  current nurse rostering code implements it:

  * KheNurseMoveAugment:  this does it, but the tasks it moves other
    than task itself are all initially unassigned, and it does not
    adapt them to r2's current timetable.

  * KheNurseOverloadAugment: this does it pretty much as described,
    but in a clumsy and probably inefficient way.

  * KheNurseUnderloadAugment: again, this does it, but using
    initially unassigned tasks only.

  * I also need to look at what the limit active intervals repairs
    do.  They are similar but not exactly.

  So I seem to be on the right track, but there is a lot more
  thinking and revising to do.

  Calls on KheTaskMoveAugment:

  * KheMoveTaskToResourceAugment, moves one task lying in tg to
    r, so it is essentially an underload augment.

  * KheOverloadAugment, moves one task currently assigned to r
    to any other resource in the task's domain.  Overload augment.

  * KheAssignResourceAugment, moves an unassigned task to any
    suitable resource.

  * KhePreferResourcesAugment, moves an inappropriate assignment
    to an suitable one.

  * KheLimitResourcesAugment, similar, only it can be fixing
    an overload or an underload.

  All of these can benefit from a generalized KheTaskMoveAugment.

7 August 2018.  Working on the new nurse rostering repairs today.
  Wrote KheTaskSequenceMoveMultiRepair, which needs an audit but
  which basically encapsulates all the new ideas.  I'm calling
  it from KheTaskMoveRepair so it's nicely integrated into the
  rest of the solver.  All audited and in good shape.

8 August 2018.  Audited calls on KheTaskMoveRepair today, and
  carefully verified that the generalized KheTaskMoveAugment is
  suitable for all these calls.  Tidied up KheLimitResourcesAugment
  a bit along the way.

  Did some testing, got right through the solving but struck this
  during testing:

     KheArchiveWrite: in event 1Sun:D of instance COI-Ikegami-3.1,
     event resource with preassigned resource B16 has task with
     missing resource assignment

  I've fixed the problem by ensuring that preassigned tasks do not
  go into the before and after task sets, since that would imply
  that they could be unassigned, or reassigned.

  Did a complete run using the new repairs on the COI instances.
  There are pluses and minuses:  on the plus side, run time has
  improved and some instances are returning better costs, notably
  COI-ERRVH where the cost has reduced from 51792 to 9588 (this
  is for KHE18x8).  But there are now four instances whose results
  are infeasible.  I guess it's back on the treadmill of analysing
  results and trying to improve them, only staying within the new
  paradigm.

14 August 2018.  The last few days I've spent writing a tutorial
  lecture for PATAT 2018.  Not sure where five days went though.
  Today I've been looking into the infeasibility thing and I've
  found that it was due to unassigning preassigned resources.
  I've basically fixed that, although it is possible that there
  are still one or two places, not often called, where it is
  being done.  However, time for a full test.

  All's well in the full test, best results so far, good
  running times (scope for more repairs, in fact), and no
  ineasibilities.  Also ran INRC1 and got pretty much the
  same results (marginally better), with some increase in
  running time but nothing terrible.

16 August 2018.  Finalized three of the four sets of overheads
  that I need for PATAT 2018 (nurse rostering, nurse history,
  and the specification tutorial), leaving just the solver
  paper overheads to go.  I've basically done them but I
  need to refresh them when I close off.

18 August 2018.  Finally got rid of the KheNurseOverload.  Did
  a full run of COI.  Results are marginally worse but I will
  stick with them for the sake of a clean software structure.

19 August 2018.  Updated the augment function for limit active
  intervals constraints.  Average results on COI are slightly
  better, both average cost and average run time, although
  some instances are doing worse.  Well worth keeping as a
  basis for futher work.

20 August 2018.  Working on GPost again, it has decayed and it
  would be good to get it back to its best.  Looking through it
  it seems that the absence of a swap function is doing the
  damage.  Some things just don't repair without a swap.

  I've done some thinking about the swap, it looks like I
  will need to imitate previous code and iterate through
  the days frame of the new resource, and for each sequence
  of sufficient length try moving the inside left and inside
  right back to old_r, or unassigning if old_r is NULL.

  But I need to get away from building a task set and then
  assigning it.  I need to assign as I move through the
  frame.

  Could I enquire of the ejector as I go as to whether
  things are looking good or not?  KheEjectorOptionalEnd?

21 August 2018.  Working on a revised KheTaskSequenceMoveRepair.

  If KheTaskMoveAugment is given a singleton resource
  group, it means that it is trying to fix an underload,
  and this should be taken into account when building
  frames - we want this resource to be not present in
  the frame that gets moved initially.  But actually this
  happens automatically.

  Written code that looks as though it will work, including
  consulting a new "es_task_seq_swaps_off" option which has
  the usual false default and turns swaps off when true.

  Posted overheads for the solver paper.  So that's all the
  overheads done now; I just need to put the stuff onto
  memory sticks.

22 August 2018.  Audited the new code that tries both
  moves and swaps.  It all looks great, ready to test.

23 August 2018.  Found a couple of small problems with
  repairs not being tried that should have been, and being
  tried when they should not have been.  Fixing them has
  brought the GPost best of 8 down to 12, which is in the
  ballpark.  It is 10 in the published conference paper.

  Worked on GPost.  Current best of 8 has cost 12, and
  after looking carefully at every defect I don't think
  repair will be able to improve it.

  I've done a general COI run and put the results in file
  khe18-08-23.pdf.  The KHE18x8 results are easily my best so far:
  optimal solutions to 8 instances, average of 1295 (conf2 was 4637),
  average running time 42.9 secs (conf2 was 60.2).

  Also a general INRC1 run, results there were almost the same
  in cost (marginally better) but there has been a disturbing
  blowout in run time; for the long+medium set from an average of
  23.7 seconds to an average of 87.8 seconds (this is for KHE18x8).

24 August 2018.  Yesterday's run on the long and medium INCR1
  instances produced (for KHE18x8) an average cost of 186 and
  an average run time of 87.8 seconds.  Just now I did another
  run, this time with option es_task_seq_swaps_off=true, and
  the result for KHE18x8 was average cost 190 and average run
  time 21.7 seconds.  I won't set this option for the sprint
  instances since they run very quickly anyway.

  Implemented code which uses swaps when workload is tight
  but not when it isn't.  Done a test and got what I wanted
  from it:  fast INRC1 run times without disturbing COI.  But
  I should implement a more accurate measure of the demand
  for resources than the one I have now.  Something based
  on the actual limit resources constraints.  It's hard
  because of the overlaps but we need a good estimate.

25 August 2018.  Worked out why CQ14-04 currently dumping core:
  domain conflicts between the tasks we are trying to group.
  Fixed it rather well by first trying the assignment, and
  then if it fails, adding the task's bounds to the leader
  task and trying again.  These bounds are then removed by
  ungrouping.  All implemented and documented.  It stopped
  the core dump.

  Did a full run of CQ14.xml, got results that were as
  good as previously, possibly better.

26 August 2018.  Last day before PATAT 2018.  I've done a
  complete new run with the current version of KHE, which I
  have labelled Version 2.3, and put the results of the runs
  and the 26 August 2018 version of the paper onto my web
  site.  Everything is there and in the right place, except
  that I have not posted a new version of KHE, for good reason:
  I don't want to be away when someone tries it and finds
  something wrong.  I'll post it when I get back.

5 September 2018.  Back at work today after returning from
  PATAT 2018.  I've identified some INRC2 instances in Andrea
  Schaerf's PATAT 2018 paper which I'll convert and add to the
  XESTT web site.  No major new projects although I plan to
  monitor the new university course timetabling competition
  and see whether I have anything to contribute there in
  the way of modelling etc.

  Read the unread portion of the PATAT 2018 proceedings.  Sent
  off an email to a student of Nysret's about overlap with KHE,
  which bounced so I've asked Nysret about that.  He's forwarded
  it, I got the email address wrong.

  Emailed jeremy.omer@insa-rennes.fr asking for INRC2 solutions.
  And also Andrea for hist.

6 September 2018.  Received solutions from Jeremy and Andrea.

8 September 2018.  Not getting much work done, not sure why,
  probably jet lag.

  I've grabbed just the directories I need:

      INRC2/hidden-dataset/xml/4-weeks/*
      INRC2/hidden-dataset/xml/8-weeks/*
      INRC2/late-dataset/xml/4-weeks/*
      INRC2/late-dataset/xml/8-weeks/*

  and written a makefile which calls the instances names we
  need.  Currently converting instance names into file names,
  within function INRC2ConvertInstanceStatic.

9 September 2018.  Made a start on a new paper for PATAT,
  "Modelling time in timetabling", in tt/patat20/pap_time.
  There are some serious implementation issues ahead.

10 September 2018.  Written the new paper down to the point
  where I have to explain how to implement the new model.
  I'll let it lie like that for a while.

11 September 2018.  Just had a brilliant time modelling
  idea:  one time (i.e. identifier) may stand for a set
  of time intervals.  Time groups are sets of times,
  that is, sets of sets of intervals.  When iterating
  over a domain, what you iterate over is the time group,
  each element of which represents a set of intervals.

  Working on extending the INRC2 model in NRConv so that
  it can convert multiple weeks.  I've done various useful
  things, possibly everything, and I'm actually generating
  archives containing vaguely correct-looking instances.
  I need to go on to solutions.

  Had a look at the solutions, both Jeremy and Andrea have
  the same format, one text file per week all in a directory
  named after the instance.

13 September 2018.  Working steadily on incr2.

14 September 2018.  Successfully producing archive files
  including solutions.  Have to audit them now and check
  their evaluations against the ones I have.

17 September 2018.  Skipped a few days owing to family troubles.
  Back at work today.  I've revised the history after values of the
  constraints (all history after is infinite in INRC2), and verified
  that all is correct except that Andrea's validator is reporting

    Optimal coverage constraints: 480

  whereas my evaluation gives 0 for this category.

  I've discovered that NRC considers the preferred and min
  bounds to be inconsistent, and is rejecting the preferred
  bounds.  Have to look into why this is; but it will be to
  do with the fact that the min bound is hard and the
  preferred bound is soft.  Fixed it and now I'm getting
  the correct cost for Andrea's one solution (1680).

  I've now confirmed that HSEval's evaluations agree with
  the validator.txt evaluations in all cases - all of Omer's
  evaluations, and the one solution that Andrea sent.

22 September 2018.  Today's job is to get INRC2 into the
  solver paper.  Done that, results are mediocre compared
  with LOR17.  I tried a 40 minute run on the 4-week
  instances.  LOR17's average cost is 2090 on the 4-week
  instances, the KHE18x8 (40 mins) average is 2665.

24 September 2018.  Skill group A_SS is often in fault, its
  members are A01, A02, A03, A04, A05, A06.  Examples of
  defective times are 2Tue:N, 2Wed:N, and 2Thu:N, constraint
  DemandConstraint:6A.

  Min 6 days between N shifts - ugly time windows implementation,
  can we do better?  Could be replaced by a limit active
  intervals constraint, min 6, with each time group being
  a negative one holding just the night shift.  There might
  be some differences in how costs are calculated, but it
  should be better.

25 September 2018.  HSEval is omitting the last day of the
  Ikegama 3.1 instance from its planning and regular timetables.
  The problem is that it has the same name as one of the Day
  time groups, because the week contains only one day.  All
  fixed now.

  Recoded "Max 6 days between day shifts" as a limit active
  intervals constraint with each day shift (negative) and a step
  cost function.  This reduced the best of 8 on COI-Ikegami-3.1
  from 58 to 39, which is good progress but still a long way
  from the optimum value, which is 2.  Cost 39 is also the
  best of 64, which is not encouraging.

  Recoded "Min 6 days between N shifts" but the best of 8
  increased from 39 to 72, which is a worry.  Not sure why,
  but I'll persevere with it for a while yet.  It ought to
  work.

  Changed KHE_MOVE_CHECKED to KHE_MOVE_EJECTING within the
  limit resources repair, got a marginal improvement from
  72 down to 69 (best of 8).

29 September 2018.  Tried grouping limit resources monitors that
  monitor the same resource type and the same set of events.
  First results on COI-Ikegami-3.1 are 64 (best of 1) and
  64 (best of 8), which is a marginal improvement on 69.
  Best of 64 is 58.

  Found a problem with the encoding of min 6 days between night
  shifts.  But after fixing it, even though it made the instance
  more relaxed, my results got worse:  138 for best of 8.  However,
  the news is not all bad.  If I could only get rid of the cost 100
  for a day off violation, I would be down to cost 38.

  Now making augment functions return true when they are successful,
  since that allows group augment functions to know when to stop
  trying.  Applying it to limit resources repairs.  Costs now are
  142 (best of 1) and 138 (best of 8).

30 September 2018.  Got best of 8 down to 37 by forcing tight
  workloads.  But can we deduce that workload limits are tight?
  Or should we just assume they are, and always do it?

  Inverted cluster constraints whose time groups are all
  negative, but that did not help COI-Ikegami-3.1, because
  its workload limit constraint does not span the whole
  cycle; it omits the first, preassigned, week.

  Did some tests (in res_seq.xml) which show pretty clearly
  that including swapping is best.  Where swapping was worse
  it was only slightly worse, and sometimes it did a lot better.
  And runtimes are slower but not much slower.

1 October 2018.  Tried calling ejection chains within resource
  matching, when there are limit resources defects.  Got a core
  dump in the group monitor handling code.

  Dumping core in code that links group monitors to child monitors.
  After extensive debugging this proved to be a case of deleting
  the same group monitor twice.  So I've inserted a check that
  when deleting a group monitor it's not equal to the most
  recently deleted group monitor.  Hopefully that will catch
  future occurrences of this problem.

  First results of ejection chains during resource matching:
  best of 1 is 24 in 9.3 seconds; best of 8 is 17 in 19.3
  seconds.  This is definitely a big cost improvement, althout
  at a significant run-time cost.  The remaining problems are
  mostly unassigned night shifts.
  
  I've tested all of COI.xml now and the code works but it
  slows things down, which is bad news for the instances that
  are already at the time limit (e.g. MER).

2 October 2018.  Did a full run today, with a 10 minute time limit.

3 October 2018.  Looking over yesterdays' full run.  It's not
  hopeless but it could be a lot better.

4 October 2018.  Verified that all of the limit resources constraints
  are in the COI instances.

  Starting to look at MER.  It has limit resources constraints but
  they do not seem to be particularly challenging.  It's a 6 week
  instance with 12 shifts per day and 54 nurses.  Not impossibly
  large, one would have thought.  Here is the summary of what you
  get with no repair:

  Summary                                       	Inf.     Obj.
  -------------------------------------------------------------------
  Avoid Unavailable Times Constraint (6 points) 	   	  110
  Cluster Busy Times Constraint (97 points) 		   	11600
  Limit Busy Times Constraint (51 points) 		   	 5590
  Limit Workload Constraint (23 points) 		   	12400
  Limit Active Intervals Constraint (144 points) 	   	44200
  Limit Resources Constraint (200 points) 		   	  453
  -------------------------------------------------------------------
      Grand total (521 points) 	   				74353

  After reducing the ejection chain solver's continue gm:

    Soln                              Defects          Cost
    -------------------------------------------------------
    AvoidUnavailableTimesMonitor           11       0.01280
    ClusterBusyTimesMonitor               236       0.15090
    LimitBusyTimesMonitor                  79       0.07900
    LimitWorkloadMonitor                   19       0.11600
    LimitActiveIntervalsMonitor           116       0.25100
    LimitResourcesMonitor                 210       0.00904
    -------------------------------------------------------
    Total                                 671       0.61874

  The result is actually better.  I need a full COI run with this
  setting, right now.

  These are the only instances that contain limit resources
  constraints:

    COI-Ozkarahan.xml
    COI-Musa.xml
    COI-Azaiez.xml
    COI-QMC-2.xml
    COI-ERMGH.xml
    COI-CHILD.xml
    COI-ERRVH.xml
    COI-HED01.xml
    COI-Ikegami-2.1.xml
    COI-Ikegami-3.1.xml
    COI-Ikegami-3.1.1.xml
    COI-Ikegami-3.1.2.xml
    COI-BCDT-Sep.xml
    COI-MER.xml

  There are 14 instances here.  Results with and without full
  soln continue gm are quite mixed, with some terrible results
  on either side.  More serious work is needed.

7 October 2018.  More or less decided to have a stack of
  time limits in KHE_OPTIONS.

8 October 2018.  Implemented and documented hierarchical time limits
  and used them to impose a 0.5 second time limit on the ejection
  chain repair part of time sweep.  First results on COI-ERMGH:

    KheGeneralSolve2018 at end (5.00 mins):
    [ Soln (instance "COI-ERMGH", diversifier 0, cost 0.01367)
      Soln                              Defects          Cost
      -------------------------------------------------------
      ClusterBusyTimesMonitor                10       0.00100
      LimitActiveIntervalsMonitor             5       0.00500
      LimitResourcesMonitor                 248       0.00767
      -------------------------------------------------------
      Total                                 263       0.01367
    ]

  We were getting 1359 long ago, so we are back to about the
  usual score.  But most recently we were getting 18204,
  because repair during time sweep was taking too much time.

  I changed the continue gm to the whole solution and tried
  best of 8.  This gave

    [ "COI-ERMGH", 8 threads, 8 solves, 8 distinct costs, 5.0 mins:
      0.01506 0.01623 0.01646 0.01660 0.01678 0.01693 0.01736 0.01745
    ]

  Changing it back to the limit monitors only gave

    [ "COI-ERMGH", 8 threads, 8 solves, 8 distinct costs, 5.0 mins:
      0.01434 0.01452 0.01468 0.01515 0.01525 0.01579 0.01581 0.01690
    ]

  which is still not great.  However, what we need now is a full
  run over COI to see where we are, generally speaking.

  Did a full run, stored in khe18-10-08.pdf.  This is using
  start_gm == limit_gm in the time sweep ejection repair.
  It's the best yet, although it would be good to get some
  further improvement on the larger instances, especially CHILD.

10 October 2018

  Changed the ejection chain solver so that it now tries swaps
  of resources at the same time when asked to.  Cost was 664,
  down to 615 owing to a small change (I am now testing that
  to_r is available before trying the move or swap), and then
  to 1111 (sadly) after implementing the full thing.  Best
  of 8 is also worse than 474, the previous best of 8:

    [ "COI-CHILD", 8 threads, 8 solves, 8 distinct costs, 154.8 secs:
      0.00631 0.00806 0.00902 0.00988 0.01028 0.01095 0.01354 0.01408
    ]

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 268.0 secs:
      0.00705 0.00706 0.00710 0.00795 0.00813 0.00884 0.01094 0.01126
    ]

  Tried grouping night and evening shifts, best of 1 was 1082,
  best of 8 is

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 270.4 secs:
      0.00588 0.00691 0.00692 0.00698 0.00797 0.00819 0.00892 0.01006
    ]

  Still not back to the 474 we had before!  Why not?  Try it
  without the new repair:

    [ "COI-CHILD", 4 threads, 8 solves, 7 distinct costs, 259.0 secs:
      0.00588 0.00589 0.00799 0.00806 0.00806 0.00889 0.00895 0.00896
    ]

  Much the same.  Setting allow_resource_swaps to true in all calls:

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 265.5 secs:
      0.00809 0.00997 0.01090 0.01096 0.01102 0.01109 0.01180 0.01193
    ]

  Hopeless.  Allowing both kinds of repair gives this:

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 298.3 secs:
      0.00464 0.00565 0.00663 0.00865 0.00871 0.00961 0.01072 0.01158
    ]

  Much better, a new record in fact.  Another slight change gives:

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 5.1 mins:
      0.00260 0.00661 0.00669 0.00766 0.00768 0.00874 0.00974 0.01280
    ]

  There's a lucky outlier here, but the average is good too.
  
  Did a full run of COI.  It's much the same as before, only a
  bit slower.  I need to think about it some more, keep working
  on COI-CHILD, do some tests on the other data sets.

11 October 2018.  Had to fix a character set problem with HSEval
  today for Gerhard.  Fixed it at home, still to upload new version.

12 October 2018.  Lost about a day to refereeing.  Back at work on
  COI_CHILD, increased the max augment nodes to 200 but it didn't help:

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 5.1 mins:
      0.00569 0.00662 0.00765 0.00768 0.00866 0.01066 0.01067 0.01572
    ]

  Here it is back at 120:

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 290.3 secs:
      0.00570 0.00669 0.00767 0.00769 0.00872 0.00961 0.00974 0.01070
    ]

  And here is another result, now sure what I did to get it:

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 287.2 secs:
      0.00472 0.00665 0.00770 0.00771 0.00869 0.00964 0.01265 0.01467
    ]

13 October 2018.  Carrying on with COI-CHILD.  I've tracked down
  the source of the variability in the results I'm getting:  it
  is due to variation in when the time limit kicks in in the
  resource matching ejection chain repair code.

  So I've tweaked things so that the initial repairs run, not for
  0.5 seconds, but for 0.5 seconds after the last improvement.
  Got this for best of 1:

    [ "COI-CHILD", 1 solution, in 5.0 mins: cost 0.00455 ]

  and this for best of 8:

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 9.8 mins:
      0.00354 0.00363 0.00459 0.00461 0.00469 0.00559 0.00756 0.00862
    ]

  Here's a second run with the same settings:

    [ "COI-CHILD", 4 threads, 6 solves, 5 distinct costs, 9.8 mins:
      0.00461 0.00471 0.00558 0.00562 0.00562 0.00669
    ]

  This is a pretty good result, but what if we bail out after,
  say, 0.2 seconds rather than 0.5 seconds?  We get this:

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 6.7 mins:
      0.00472 0.00558 0.00663 0.00665 0.00760 0.00762 0.00767 0.00768
    ]

  It's faster but not as good.  How repeatable is it now?

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 6.5 mins:
      0.00469 0.00556 0.00558 0.00662 0.00669 0.00766 0.00771 0.00968
    ]

  Not particularly repeatable, it seems, at least not at 0.2 seconds.
  Now what happens if we abandon the main repair if there is no
  improvement after 20 seconds?

    [ "COI-CHILD", 4 threads, 7 solves, 7 distinct costs, 9.9 mins:
      0.00354 0.00460 0.00553 0.00565 0.00575 0.00664 0.00671
    ]
    [ "COI-CHILD", 4 threads, 7 solves, 7 distinct costs, 9.9 mins:
      0.00354 0.00363 0.00459 0.00461 0.00462 0.00559 0.00562
    ]

  I'm doing each run multiple times because of the unpredictability.
  Now if there is no improvement after 10 secs:

    [ "COI-CHILD", 4 threads, 5 solves, 4 distinct costs, 9.9 mins:
      0.00461 0.00462 0.00462 0.00566 0.00665
    ]
    [ "COI-CHILD", 4 threads, 7 solves, 7 distinct costs, 9.9 mins:
      0.00267 0.00354 0.00363 0.00461 0.00462 0.00559 0.00562
    ]

  This suggests that the ejection chain solver is still finding
  improvements right up until the last 10 seconds of its run.
  Can that be right?  Do a single run and see.

15 October 2018.  First results from time limits on phases:

    [ "COI-CHILD", 1 solution, in 268.1 secs: cost 0.00369 ]

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 9.5 mins:
      0.00457 0.00462 0.00555 0.00672 0.00761 0.00767 0.00863 0.01064
    ]

    [ "COI-CHILD", 4 threads, 8 solves, 7 distinct costs, 9.3 mins:
      0.00357 0.00359 0.00467 0.00556 0.00556 0.00564 0.00657 0.00756
    ]

  Changed gettimeofday to clock_gettime and CLOCK_THREAD_CPUTIME_ID.
  It seems to be working, but we need wall clock time for reporting.

    [ "COI-CHILD", 1 solution, in 237.5 secs: cost 0.00868 ]
    [ "COI-CHILD", 1 solution, in 268.3 secs: cost 0.00261 ]
    [ "COI-CHILD", 1 solution, in 260.7 secs: cost 0.00678 ]
    [ "COI-CHILD", 1 solution, in 244.8 secs: cost 0.00559 ]

  There is a lot of variation even with the new method, so I will
  probably go back to the old method.

  I also got some output from KheResourceRematch, which showed that
  it was not doing very much.  There was one improvement by 100 and
  a dozen or so by 1, not really worth the time it was taking.  Here
  we are again without rematching:

    [ "COI-CHILD", 1 solution, in 112.5 secs: cost 0.00860 ]
    [ "COI-CHILD", 1 solution, in 98.9 secs: cost 0.00867 ]
    [ "COI-CHILD", 1 solution, in 98.7 secs: cost 0.00761 ]
    [ "COI-CHILD", 1 solution, in 101.1 secs: cost 0.00971 ]

  It does seem to be generally worse without rematching.  And
  rematching does do interesting stuff that no-one else does.
  Another go with rematching; I observed it removing several
  cost 100 defects:

    [ "COI-CHILD", 1 solution, in 274.1 secs: cost 0.00359 ]

  But we don't want to crowd out ejection chain repairs either.
  Here's a run where at most half the time given to each repair
  phase is given to rematch:

    [ "COI-CHILD", 1 solution, in 247.0 secs: cost 0.00265 ]
    [ "COI-CHILD", 1 solution, in 253.1 secs: cost 0.00657 ]
    [ "COI-CHILD", 1 solution, in 236.0 secs: cost 0.00461 ]
    [ "COI-CHILD", 1 solution, in 267.7 secs: cost 0.00363 ]

  These do seem to be better on the whole.  Time for best of 8:

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 9.7 mins:
      0.00358 0.00464 0.00659 0.00662 0.00758 0.00768 0.00774 0.00855
    ]
    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 10.2 mins:
      0.00464 0.00467 0.00657 0.00755 0.00760 0.00768 0.00863 0.01166
    ]
    [ "COI-CHILD", 4 threads, 8 solves, 7 distinct costs, 9.1 mins:
      0.00359 0.00359 0.00461 0.00557 0.00657 0.00666 0.00768 0.00856
    ]

  One solution, no time limits at all:

    [ "COI-CHILD", 1 solution, in 5.9 mins: cost 0.00553 ]
    [ "COI-CHILD", 1 solution, in 5.9 mins: cost 0.00553 ]

  Sure enough, it's repeatable.  Best of 8:

    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 14.0 mins:
      0.00370 0.00455 0.00456 0.00457 0.00553 0.00566 0.00657 0.00869
    ]
    [ "COI-CHILD", 4 threads, 8 solves, 8 distinct costs, 14.1 mins:
      0.00370 0.00455 0.00456 0.00457 0.00553 0.00566 0.00657 0.00869
    ]

  Once again, repeatable.  Time for a full run of COI with time limits.

16 October 2018.  Had a look at my new COI results.  They are the
  best yet, except that running times are excessive for some of
  the larger instances.  Changed the paper to include a discussion
  of KHE18's time limits.

  Found out why COI-MER was taking too long, it was that
  KheTimeSweepAssignResources (c) had no time limit.
  I've added this now and am re-running the COI tests.
  Here is one basic one which shows that we are in the
  ballpark now.  NB MER is a six-week instance.

    [ "COI-MER", 1 solution, in 12.7 mins: cost 0.11319 ]

  Here it is with the latest version of the time limit:

    [ "COI-MER", 1 solution, in 5.8 mins: cost 0.13150 ]

  The time limit is working but the result is not great.  Still,
  this we want a 5 minute time limit for six-week instances.

  Fiddled with KheResourceRematch to make it able to vary its
  behaviour on different calls.  All done and documented.  The
  cost came out somewhat worse:

    [ "COI-MER", 1 solution, in 5.8 mins: cost 0.14203 ]

  But still it seems like a good idea in principle.  Now
  I've done another full COI run, in khe18-10-16.pdf.  I've
  got decent run times with some deterioration in cost.

  Wrote up resource rematching in the paper.  For some reason I
  didn't include it before.

  Done the INRC2 run, saved in khe18-10-17.pdf.  The INRC2
  results have improved, but the conclusion remains the same:
  the 4 week results are competitive, the 8 week results not.

18 October 2018.  Fixed a small time limit bug; the daily redo
  was being given the full daily limit, not half of it.

  A long (30 minutes per instance) INRC2 run finished this
  morning.  Again, the same conclusion:  the 4 week results
  are competitive, the 8 week results are not.

19 October 2018.  Finished the new timer module and have a
  clean compile using it all.  Not tested but should work.

  Started a CQ14 run.  Got to

    [ "CQ14-24", 4 threads, 8 solves, 8 distinct costs, 8.3 mins:
      1635.78195 1657.79588 1711.76958 1711.78940
      1725.79384 1726.77974 1756.77521 1804.79408
    ]
    Killed

20 October 2018.  Trying to fix yesterday's memory problems.
  I've now removed all monitor frames, including the ones
  made when checking workload limits which may have been the
  source of the problems.  Started another full CQ14 run.
  It actually ran to completion, so that's a problem solved.

21 October 2018.  Worked on writing out time limit tests,
  have to work on reading them in again now.  It needs some
  careful design.

  A general mechanism for storing inside a solution what is
  needed to regenerate that solution would be good.  But is
  it a bridge too far just now?  Include it in a later
  format perhaps.  Anyway I've decided not to carry on
  with time limit storing for the time being.

23 October 2018.  Commented out the time limit writing code.

24 October 2018.  Started work on the changes to the ejection
  chain solvers to bring them into line with the description
  in the nurse rostering solver paper.

    to_r == NULL

      from_r == NULL:  impossible
      from_r != NULL:  unassigning; find other tasks to assign to from_r

    to_r != NULL && TimesFree(to_r)

      from_r == NULL: assigning; look for other tasks to unassign
      from_r != NULL: moving; look for other tasks to assign to from_r

    to_r != NULL && !TimesFree(to_r)

      from_r == NULL: assigning; look for other tasks to unassign
      from_r != NULL: moving:  can really only swap on the spot.

  Although if we worked out the number of tasks in the imbalance
  we could try moving that many the other way.  Any that get
  bumped count towards the number.  If there are still some
  required, we go searching along to_r's timetable.  But we
  always move consecutive sequences.

  Get some number of consecutive from_r tasks, or gaps, and
  get as many consecutive tasks as possible from to_r.

25 October 2018.  Why are we moving?

  (1) To increase the number of resources from a given
      resource group rg that lie in some set of tasks:

	for each task in the set of tasks not assigned rg:
          KheTaskMoveAugment(ej, task, rg, NULL, false)

      Here task may initially be assigned NULL.  It is not
      initially assigned a resource from rg.  If some other
      task is unassigned by the move, we don't want that to
      be in the set of tasks.

  (2) To decrease the number of resources from a given
      resource group rg that lie in some set of tasks:

	for each task in the set of tasks assigned rg:
          KheTaskMoveAugment(ej, task, domain(task), rg, true)

      Here task cannot initially be assigned NULL.  It is
      initially assigned a resource from rg.  If some other
      task is assigned by the move, we don't want that to be
      in the set of tasks.

  (3) To increase the number of times from a given time
      group tg that a given resource r is busy:

	 for each task at each time of tg when r is free:
           KheTaskMoveAugment(ej, task, single(r), NULL, false)

      Here task may initially be assigned NULL.
      (KheMoveTaskToResourceAugment)

  (4) To decrease the number of times from a given time
      group tg that a given resource r is busy:

	 for each task at each time of tg initially assigned r:
           KheTaskMoveAugment(ej, task, domain(task), NULL, true)

      Here task is initially assigned r, cannot initially be assigned
      NULL.  (KheOverloadAugment)

  (5) To reduce to 0 the number of times from a given time
      group that a given resource r is busy:

	 for the tasks ts at the times of tg initially assigned r:
           KheTaskSetMoveAugment(ej, ts, domain(ts[0]), NULL, true)

      Here ts is initially assigned r, cannot initially be assigned
      NULL.  (KheOverloadAugment)

  At present, when we find unassigned tasks, we have no control over
  whether they are day, afternoon, or night.  In fact they will always
  be the first encountered.  We should at least start with the same
  time index in the time group as the one that the starting task has.
  Or maybe we should not widen when the initial task is unassigned.

  KheResourceMoveRepair, called from KhePreferResourcesAugment, is
  in the mix.  It's doing the kind of swap we are wondering about.

  Here's a proposal:  if from_r is NULL, or to_r is NULL, do as much
  as makes sense but no more.  The main case is from_r != NULL and
  to_r != NULL, and swapping the timetables of from_r and to_r on the
  day of task (or the days of ts) will satisfy the immediate need:

  Swap the timetables of from_r and to_r on the day of task and on
  a total of 0, 1, 2, or 3 adjacent days.  Then if the imbalance
  is X, search for up to N other places in their timetables where 
  an imbalance of X can be corrected by moving up to K adjacent
  days, and try that as part of the repair.  But also try nothing.

  Wrote up the new structure in the solver paper.  It should be
  fairly easy to implement.  Doing that is next.

26 October 2018.  Revised the new repair structure in the solver
  paper, it's ready to implement now.

27 October 2018.  I've finished the new KheTaskMoveAugment, except
  not the double swaps yet.  I need to audit everything and test
  it as it is.

28 October 2018.  In KheMoveTaskToResourceAugment, which is
  called by KheUnderloadAugment, I've implemented this: "Can
  the solver be told when two event resources are equivalent?
  In that case if both are unassigned it can try assigning only
  one of them."  But there may be other places to try it.

  Enhanced KheTaskMoveAugment to allow meet moves, and thus was
  able to get rid of KheOldTaskMoveAugment.  This in turn made
  a huge whack of old code unused, so I commented all that out.

  Found that KHE_ACTIVE_AUGMENT_TYPE was not being used any
  more, so I got rid of it.

  Audited yesterday's work carefully.  All done and ready to test.
  Did a first test of COI.xml.  Got rid of a few little bugs and
  it's all working, but the results are not very good.  I guess
  the next step is double day-set swaps, then back to the grind.

29 October 2018.  Implemented double day-set swaps, then did a
  full COI test, which did not do well.  Did some thinking about
  the repairs, prompted by COI-QMC-1.xml, which produced a rewrite,
  which is all done with a clean compile, but not tested yet.

30 October 2018.  Rewrite of day-set swaps audited and ready to test:

    [ "COI-QMC-1", 4 threads, 8 solves, 5 distinct costs, 25.0 secs:
      0.00019 0.00019 0.00019 0.00020 0.00020 0.00021 0.00024 0.00025
    ]

  This is the same cost I have traditionally been getting, although
  the run time is slower.  Tried a full COI run, it went very slowly,
  I've tracked it down to too many alternative repairs.  I put a time
  limit check into RepairEnd, and got the time under control:

    [ "COI-ERRVH", 1 solution, in 5.1 mins: cost 0.58052 ]

  But the cost here is still no good at all, I had it down to
  2810 before, which was competitive with 2001, the best known.
  HSEval's evaluation shows that all kinds of constraints are
  contributing to the poor cost.  In short it is a mess.

  Tried distributing the time better between the parts of
  rematching:

    [ "COI-ERRVH", 1 solution, in 5.1 mins: cost 0.51301 ]

  Only a marginal improvement.

31 October 2018.  Decided to work on submitting the journal
  versions of the first two papers, and give solving a rest.
  I might not submit the solver paper, depending on where I
  am on 7 November.

1 November 2018.  Installed a new version of the XESTT web
  pages, including links to INRC2-4.xml and INRC2-8.xml.
  Submitted modelling paper and history paper to AOR.

2 November 2018.  Added a filter to KheTaskMoveAugment so
  that it only tries repairs that suit the reason why the
  function was called.

    [ "COI-QMC-1", 4 threads, 8 solves, 4 distinct costs, 3.8 secs:
      0.00027 0.00027 0.00028 0.00029 0.00031 0.00031 0.00031 0.00031
    ]

  This is the right running time, which is good, but it is not
  competitive with the best of 8 I was getting before (19).
  I've just started looking into this, most of the defects
  are failed time on requests.

    [ "COI-CHILD", 1 solution, in 5.0 mins: cost 0.03237 ]

  Not as bad as it was, but still not competitive with my
  previous (763).

    [ "COI-GPost", 1 solution, in 1.0 secs: cost 0.00016 ]

    [ "COI-GPost", 4 threads, 8 solves, 3 distinct costs, 3.0 secs:
      0.00016 0.00016 0.00022 0.00026 0.00026 0.00026 0.00026 0.00026
    ]

  Again, these are inferior to what I had before (12 and 10).  The
  defects in the cost 16 solution seem to be quite intractable.

4 November 2018.  Tried various changes to the randomization in
  the resource matcher.  Nothing was systematically better.  At
  times I did get the cost 12 solution, it seems to be an outlier
  that I was lucky to get, before.  It looks like getting below
  16 is very hard indeed.

5 November 2018.  Had an idea:  focus the limited number of
  opposite direction swaps on parts of the cycle close to
  the starting point.  It gives this:

    [ "COI-GPost", 4 threads, 64 solves, 8 distinct costs, 22.5 secs:
      0.00012 0.00012 0.00012 0.00016 0.00016 0.00016 0.00016 0.00016
      0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00016
      0.00019 0.00019 0.00022 0.00022 0.00022 0.00023 0.00023 0.00023
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00034 0.00034
      0.00034 0.00035 0.00035 0.00035 0.00035 0.00035 0.00035 0.00035
    ]

  as compared with not focussing on nearby positions:

    [ "COI-GPost", 4 threads, 64 solves, 8 distinct costs, 24.8 secs:
      0.00012 0.00012 0.00012 0.00016 0.00016 0.00016 0.00016 0.00016
      0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00016
      0.00016 0.00019 0.00019 0.00022 0.00022 0.00022 0.00023 0.00023
      0.00023 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00034
      0.00034 0.00035 0.00035 0.00035 0.00035 0.00035 0.00035 0.00035
    ]

  There are some differences but not significant ones.  This is
  what happens when you turn off checking the reason:

    [ "COI-GPost", 4 threads, 64 solves, 11 distinct costs, 30.6 secs:
      0.00010 0.00010 0.00010 0.00010 0.00010 0.00010 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00013 0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00017
      0.00017 0.00018 0.00018 0.00018 0.00020 0.00020 0.00021 0.00021
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00030 0.00030 0.00030
    ]

  It's slower, but wow.  Best of 8:

    [ "COI-GPost", 4 threads, 8 solves, 6 distinct costs, 5.3 secs:
      0.00010 0.00012 0.00016 0.00016 0.00018 0.00022 0.00022 0.00030
    ]

  Looks like reasons don't matter.  Using nearby swaps:

    [ "COI-GPost", 4 threads, 8 solves, 6 distinct costs, 7.0 secs:
      0.00010 0.00012 0.00016 0.00016 0.00018 0.00022 0.00022 0.00030
    ]

  Same solutions but took longer to find.  Setting max_days = 6:

    [ "COI-GPost", 4 threads, 8 solves, 7 distinct costs, 13.7 secs:
      0.00009 0.00011 0.00012 0.00022 0.00022 0.00023 0.00027 0.00031
    ]

  Best of 64 with max_days = 6:

    [ "COI-GPost", 4 threads, 64 solves, 19 distinct costs, 98.0 secs:
      0.00009 0.00009 0.00009 0.00011 0.00011 0.00011 0.00011 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00013 0.00016 0.00016 0.00018 0.00018
      0.00019 0.00020 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00023
      0.00023 0.00023 0.00027 0.00027 0.00027 0.00030 0.00031 0.00031
      0.00032 0.00032 0.00033 0.00033 0.00040 0.00041 0.00214 0.00428
    ]

  Setting max_days = 5, which might be a good compromise:

    [ "COI-GPost", 4 threads, 8 solves, 6 distinct costs, 11.4 secs:
      0.00010 0.00010 0.00010 0.00011 0.00012 0.00013 0.00022 0.00042
    ]

  Now what about ensuring that each double swap is at the end of a
  run?  When not swapping nearby:

    [ "COI-GPost", 4 threads, 8 solves, 6 distinct costs, 9.5 secs:
      0.00010 0.00010 0.00010 0.00011 0.00012 0.00013 0.00022 0.00042
    ]

  and when swapping nearby:

    [ "COI-GPost", 4 threads, 8 solves, 5 distinct costs, 9.8 secs:
      0.00010 0.00010 0.00010 0.00011 0.00012 0.00022 0.00022 0.00042
    ]

  This seems to be just as good in cost, and a bit faster, which
  suggests that most good swaps are at the ends of runs.  Best of
  64 (swapping nearby, max_days = 5):

    [ "COI-GPost", 4 threads, 64 solves, 14 distinct costs, 55.5 secs:
      0.00009 0.00009 0.00010 0.00010 0.00010 0.00010 0.00010 0.00010
      0.00010 0.00010 0.00010 0.00010 0.00011 0.00011 0.00011 0.00011
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00014
      0.00014 0.00014 0.00014 0.00016 0.00016 0.00017 0.00017 0.00019
      0.00019 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00025 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026
      0.00026 0.00031 0.00031 0.00031 0.00042 0.00042 0.00042 0.00217
    ]

  Best of 64, not swapping nearby, max_days = 5:

    [ "COI-GPost", 4 threads, 64 solves, 14 distinct costs, 52.8 secs:
      0.00009 0.00009 0.00010 0.00010 0.00010 0.00010 0.00010 0.00010
      0.00010 0.00010 0.00010 0.00010 0.00011 0.00011 0.00011 0.00011
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00013
      0.00013 0.00013 0.00014 0.00014 0.00014 0.00014 0.00016 0.00016
      0.00016 0.00017 0.00017 0.00019 0.00019 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00025 0.00026 0.00026
      0.00026 0.00026 0.00031 0.00031 0.00031 0.00042 0.00042 0.00042
    ]

  Not swapping nearby seems to be marginally better.  Best of 64, not
  swapping nearby, max_days = 4:

    [ "COI-GPost", 4 threads, 64 solves, 10 distinct costs, 24.3 secs:
      0.00009 0.00009 0.00009 0.00010 0.00010 0.00010 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00013 0.00016 0.00016 0.00016 0.00016
      0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00017 0.00017
      0.00021 0.00021 0.00021 0.00021 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00030 0.00030 0.00030
    ]

  It's faster, obviously, but curiously enough it is also better.

  Final experiments
  -----------------

  Here are my final experiments on COI-Post, with max_days = 4,
  nearby = false or true, ends of runs only = false or true.

  max_days = 4, nearby = false, ends = false:

    [ "COI-GPost", 4 threads, 64 solves, 11 distinct costs, 29.2 secs:
      0.00010 0.00010 0.00010 0.00010 0.00010 0.00010 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00013 0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00017
      0.00017 0.00018 0.00018 0.00018 0.00020 0.00020 0.00021 0.00021
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00030 0.00030 0.00030
    ]

  max_days = 4, nearby = false, ends = true:

    [ "COI-GPost", 4 threads, 64 solves, 10 distinct costs, 23.4 secs:
      0.00009 0.00009 0.00009 0.00010 0.00010 0.00010 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00013 0.00016 0.00016 0.00016 0.00016
      0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00017 0.00017
      0.00021 0.00021 0.00021 0.00021 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00030 0.00030 0.00030
    ]

  max_days = 4, nearby = true, ends = false:

    [ "COI-GPost", 4 threads, 64 solves, 11 distinct costs, 30.5 secs:
      0.00010 0.00010 0.00010 0.00010 0.00010 0.00010 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00013 0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00017
      0.00017 0.00018 0.00018 0.00018 0.00020 0.00020 0.00021 0.00021
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00030 0.00030 0.00030
    ]

  max_days = 4, nearby = true, ends = true:

    [ "COI-GPost", 4 threads, 64 solves, 9 distinct costs, 22.9 secs:
      0.00010 0.00010 0.00010 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00013 0.00016 0.00016 0.00016 0.00016
      0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00017 0.00017
      0.00021 0.00021 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00030 0.00030 0.00030
    ]

  It's clear from this that ends = false produces longer run times
  for no improvement in cost.  So we adopt ends = true:

  max_days = 4, nearby = false, ends = true:

    [ "COI-GPost", 4 threads, 64 solves, 10 distinct costs, 23.4 secs:
      0.00009 0.00009 0.00009 0.00010 0.00010 0.00010 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00013 0.00016 0.00016 0.00016 0.00016
      0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00017 0.00017
      0.00021 0.00021 0.00021 0.00021 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00030 0.00030 0.00030
    ]

  max_days = 4, nearby = true, ends = true:

    [ "COI-GPost", 4 threads, 64 solves, 9 distinct costs, 22.9 secs:
      0.00010 0.00010 0.00010 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00013 0.00016 0.00016 0.00016 0.00016
      0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00017 0.00017
      0.00021 0.00021 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00026 0.00026 0.00026
      0.00026 0.00026 0.00026 0.00026 0.00026 0.00030 0.00030 0.00030
    ]

  There is little to choose between these two (best of 8 for nearby
  = false is 10), but I will go for nearby = false because it
  increases coverage.  Perhaps we need adjacent plus not nearby?
  Actually this would remove one defect from the current best of 8.

6 November 2018.  Implemented the adjacent and far idea and got this:

    [ "COI-GPost", 4 threads, 8 solves, 6 distinct costs, 9.5 secs:
      0.00009 0.00016 0.00016 0.00016 0.00022 0.00023 0.00026 0.00030
    ]

    [ "COI-GPost", 4 threads, 64 solves, 13 distinct costs, 42.7 secs:
      0.00009 0.00009 0.00009 0.00009 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00015 0.00015 0.00015
      0.00015 0.00016 0.00016 0.00016 0.00016 0.00016 0.00016 0.00016
      0.00017 0.00017 0.00021 0.00021 0.00022 0.00022 0.00022 0.00022
      0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00022 0.00023
      0.00023 0.00026 0.00026 0.00026 0.00026 0.00026 0.00026 0.00028
      0.00028 0.00030 0.00030 0.00030 0.00035 0.00035 0.00035 0.00039
    ]

  This is as good as COI-GPost is likely to get.

  Started a full COI run.  Pretty disastrous on the INRC1 instances.
  The problem was that the requested code was going mad, assigning
  clashes.  It knew nothing about the day frame!  Fixed, things are
  a lot better now.

7 November 2018.  Fixed the clashes problem, it was caused by
  grouped tasks.  First result after the fix:

    [ "INRC1-L02", 1 solution, in 142.4 secs: cost 0.00238 ]

  The cost is good, the running time is not.  Got a few more
  clashes in COI-HED01, and lo and behold when I fixed them
  I got what might be a new best cost (I checked a couple of
  older PDFs and they had 154, even for KHE18x8):

    [ "COI-HED01", 1 solution, in 91.3 secs: cost 0.00149 ]

  although once again the running time is not good.

9 November 2018.  Did nothing at all yesterday.  Today I have
  analysed the clashes problem and implemented a solution for
  swapping (not yet for assigning and unassigning), and got

    [ "COI-BCV-3.46.2", 4 threads, 8 solves, 3 distinct costs, 23.4 secs:
      0.00896 0.00896 0.00897 0.00897 0.00897 0.00897 0.00897 0.00898
    ]

  which is in the ballpark.  In the conference version of the paper,
  KHE18x8 also go 896, but in 10.1 seconds, which is about half the
  time.  This is with the debug checks turned off, so it is a genuine
  loss of efficiency that I will need to work on, some time.

  Finished a pretty full run, see khe18-11-09.pdf.

10 November 2018.  Looked over khe18-11-09.pdf, the results are
  slower and mostly worse, although there are some better ones.
  The general impression is that the extra running time is
  helping some smaller instances but clobbering larger ones.
  I need to find a smaller set of repairs which is just as
  effective but runs faster.

11 November 2018.  Updated nrconv to recode complete weekends in
  COI-ERRVH.  Yesterday KH18 produced 46635, today it's worse:

    [ "COI-ERRVH", 1 solution, in 5.1 mins: cost 0.47827 ]

    [ "COI-ERRVH", 4 threads, 8 solves, 8 distinct costs, 10.2 mins:
      0.38065 0.39876 0.40084 0.41470 0.42274 0.43008 0.44067 0.46707
    ]

  Fixed up a complete weekend constraint that didn't convert and got

    [ "COI-ERRVH", 4 threads, 8 solves, 8 distinct costs, 10.2 mins:
      0.40141 0.41341 0.41816 0.45150 0.45451 0.45973 0.46720 0.48109
    ]

  Even worse!  Anyway this has to be our starting point.

18 November 2018.  For the last week I have been too busy on
  various jobs to do any timetabling work.  Sadly, there will
  probably be more of that between now and Christmas.

  COI-ERRVH best soln in archive has cost 2001, I was previously
  getting 3873 (best of 1) and 2810 (best of 8), but currently
  48517 (best of 1) and 0.40141 (best of 8).  After doing some
  recoding (at most 1 consecutive free weekend), even worse:

    [ "COI-ERRVH", 1 solution, in 5.1 mins: cost 0.49607 ]

  Before requiring the swap or unassign interval to be busy:

    [ "COI-ERRVH", 1 solution, in 5.1 mins: cost 0.46401 ]

  After requiring the swap or unassign interval to be busy:

    [ "COI-ERRVH", 1 solution, in 5.1 mins: cost 0.44345 ]

  So it's a bit better but a lot is going wrong somewhere.
  After grouping limit resources monitors by frame, no change.
  Are they even getting repaired, I wonder?

3 December 2018.  Big gap caused by various things, mainly
  family duties.  Hopefully things will be quieter now.
  Removed slices and polarities from frames.  All done and
  documented.  Will be important later when I implement a
  new time model.

4 December 2018.  Removed KHE_MOVE_REASON, since it did no
  good before.  Done a skeleton version of the new repair
  function I want to try.  It will only swap all busy and
  all free, and it will swap a number of tasks, not a number
  of days.

5 December 2018.  Simplified the task set moving interface
  by building one task set for everything that needs to move.
  It might actually save time given that the set is used
  many times when double moving.  Kept working and now have
  clean compile of new task move augment code.

5 December 2018.  New KheTaskMoveAugment implemented and
  audited, and I've begun testing.  Found and fixed a bug
  that was causing clashes, then got this:

    [ "COI-GPost", 1 solution, in 0.7 secs: cost 0.00012 ]

  But when I tried best of 8 I got a clash again, this
  time on a double move.  That will be because NextRun
  does not take the full extent of the task into account.

6 December 2018.  Tested the new KheTaskMoveAugment.  It is
  producing clashes, I've started a rewrite now.

7 December 2018.  Finished rewriting KheTaskMoveAugment to avoid
  clashes.  First results:

    [ "COI-GPost", 1 solution, in 0.3 secs: cost 0.00012 ]

    [ "COI-GPost", 4 threads, 8 solves, 2 distinct costs, 1.1 secs:
      0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00012 0.00017
    ]

  This seems pretty good, although it would be even better if
  we could get below 12.  In the conference paper KHE18x8 got 10.
  Doing a COI run now, to see how the land lies.

  Got more clashes when solving COI-BCV-3.46.2.xml, have to
  look into those now.

    [ "COI-BCV-3.46.2", 4 threads, 8 solves, 2 distinct costs, 9.3 secs:
      0.00896 0.00896 0.00896 0.00896 0.00896 0.00896 0.00897 0.00897
    ]

  These are very good results, both in cost and in time.  Did
  a full COI run and got results very close to my previous best
  results (18-08-24), good enough to build on from here.  One
  obvious instannce to work on now is COI-QMC-1, whose best
  soln has cost 13, for which I previously got 26 but am
  currently getting 8036.

  Started work on COI-QMC-1.  The problem is time on requests not
  being fulfilled, i.e. cluster busy times underloads.  No repairs
  are tried, because there is only one to_r (the underloaded
  resource) and for that resource, KheTaskSetAllTasksAtFreeTimes
  fails.

8 December 2018.  Worked on the solve paper today.  It's in good
  shape but it does not describe what to do about task moves that
  fail because the resource is already busy at that time.

9 December 2018.  Thinking about task moves that fail because the
  resource is busy at the time.  The calls on KheTaskSetMoveAugment
  and KheTaskMoveAugment fall into three classes:

  * KheMoveTaskToResourceAugment; this is the one for which rg is
    a singleton resource group.  It is called by KheUnderloadAugment.
    We might want to implement KheMoveTaskToResourceAugment differently,
    that is without calling KheTaskMoveAugment.

  * KheOverloadAugment; this is the conventional use, for which
    KheTaskSetMoveAugment and KheTaskMoveAugment are fine as is.

  * KheAssignResourceAugment, KhePreferResourcesAugment, and
    KheLimitResourcesAugment; here the aim is to fix the task,
    not a resource, and so there will usually be several resources
    to choose from and all is well.

  It is only the first one, KheMoveTaskToResourceAugment, where
  there is a problem.

10 December 2018.  Got to a point where I'm ready to implement the
  alternative to task moves that fail because the resource is busy
  at the time.  Have a good design, documented in the PATAT paper.
  It replaces the two calls to KheMoveTaskToResourceAugment.

11 December 2018.  Implementing the new repairs today.  Done a lot
  of good stuff, more still to do.

12 December 2018.  Still implementing the new repairs.  All done
  except expansion of one case.  Without that, here are some
  results.  The old repairs gave

    [ "COI-QMC-1", 1 solution, in 0.5 secs: cost 0.11044 ]

  Here are the first results with the new repairs:

    [ "COI-QMC-1", 1 solution, in 0.8 secs: cost 0.03039 ]

    [ "COI-QMC-1", 4 threads, 8 solves, 8 distinct costs, 2.1 secs:
      0.00042 0.01036 0.01037 0.01040 0.01041 0.01043 0.01045 0.03039
    ]

  In the conference paper, KHE18 has cost 52 (0.8 secs), KHE18x8 has
  cost 46 (2.9 secs), and the best soln cost is 13.
  In khe18-08-24.pdf, KHE18 has cost 35 and KHE18x8 has cost 32.
  So we are back in the ballpark but not quite back to our best.
  We need to look into the poorer results.  But perhaps implement
  expansion first.

14 December 2018.  Implemented expansion now.  First results:

    [ "COI-QMC-1", 1 solution, in 0.8 secs: cost 0.01060 ]

  and

    [ "COI-QMC-1", 4 threads, 8 solves, 8 distinct costs, 2.2 secs:
      0.00050 0.00051 0.01045 0.01047 0.01052 0.01060 0.01062 0.01066
    ]

  Pity it seems to be worse, but we can get to work from here.  Best
  of 32 includes a cost 42 solution:

    [ "COI-QMC-1", 4 threads, 32 solves, 29 distinct costs, 8.5 secs:
      0.00042 0.00047 0.00048 0.00049 0.00050 0.00050 0.00051 0.00051
      0.00055 0.00058 0.00061 0.01044 0.01045 0.01047 0.01049 0.01050
      0.01052 0.01054 0.01055 0.01058 0.01060 0.01061 0.01062 0.01063
      0.01065 0.01066 0.01066 0.01067 0.02049 0.02051 0.02060 0.02068
    ]

  but this is still not as good as the conf2 result of 32.

  The problem with the doit1 solution is that resource E has two night
  shifts in a row when one of them is supposed to be an O.  The only
  repair that applies right now is replacing the N by an O, but then
  the second N is one day's work isolated, plus there is the unassigned
  night shift, so this repair goes nowhere.

  Found and fixed a bug that was in effect turning off time on
  requests.  After that:

    [ "COI-QMC-1", 1 solution, in 0.6 secs: cost 0.02026 ]

    [ "COI-QMC-1", 4 threads, 8 solves, 8 distinct costs, 1.9 secs:
      0.01036 0.01040 0.02025 0.02026 0.02028 0.02032 0.02034 0.04019
    ]

  Not great, but it's progress of a kind.  The two problems in the
  0.02026 solution are two unassigned tasks on a day when every
  resource is assigned, although many of the assignments are to
  task O.  But KheTaskMoveAugment would naturally find no moves to
  make in this situation.  I need to give assigned resources a
  chance as well.  Does this mean unifying with KheUnderloadAugment?

15 December 2018.  Added more repairs to assign resource augments, got

    [ "COI-QMC-1", 1 solution, in 0.7 secs: cost 0.00027 ]

    [ "COI-QMC-1", 4 threads, 8 solves, 6 distinct costs, 1.6 secs:
      0.00020 0.00020 0.00021 0.00022 0.00024 0.00026 0.00027 0.00027
    ]

  These are my best results so far.  The best known result has cost 13,
  so this is probably good enough.

16 December 2018.  Working on eliminating clashes from the new code.
  I've fixed some and HED01 is running right through now:

    [ "COI-HED01", 1 solution, in 21.6 secs: cost 0.00294 ]

  Not a great solution, and the run time is far worse than I got
  in conf2_khe18.pdf.  But still it's worth a full run now.

  The full run went very well.  The results for KHE18x8 were
  average cost 1141 and average run time 99.7 seconds.  This
  compares well with the khe18-08-24.pdf results, which have
  average cost 1295 and average run time 41.7 seconds.  The
  longer average run time is essentially about KHE18 running
  for 5 minutes on the four hard instances, as now planned.
  The average cost result is a significant improvement: 12%.
  There are also 11 optimal results, up from 9 in khe18-08-24.pdf.
  In fact, looking down the individual results there are none
  at all that stand out as requiring further work.

  Also tried the INRC1 instances.  The results are slightly
  better than khe18-08-24.pdf, which is good, but run times
  are about four times longer, which is hardly worth it.  So
  it seems I need to do something to reduce run times without
  worsening the results I'm getting.

  Also tried the CQ14 instances.  Results are quite good,
  similar to my best results from before.  More work needed.

  Also tried the INCR2 instances.  They've improved too,
  although the general conclusion about them (that the 4-week
  solutions are competitive and the 8-week are not) still holds.

  All saved in khe18-12-16.pdf.

17 December 2018.  Looked through yesterday's results, decided
  to play with COI-BCDT-Sep for a while.  I've been studying it:

    1  Vacation
    2  Morning
    3  Afternoon
    4  Night

    All of the constraints have weight 1 or 10, except that
    "At least 10 days off" has weight 250.  In the current
    solution, all the violations have weight 10.

    "Try to assign nights in blocks of three (although a four
    block is allowed if otherwise there is a night before a
    free saturday)" contributes 40 to the current solution cost.
    What it actually means is this:

      "You can't have a night shift on Friday night and also a
      free day (including a vacation day) on the Saturday":
      <ClusterBusyTimesConstraint Id="Constraint:3">
	<AppliesToTimeGroup Reference="1-4Fri1"/>
	<TimeGroups>
	  <TimeGroup Reference="1Fri4"/>
	  <TimeGroup Reference="1Sat2-4" Polarity="negative"/>
	</TimeGroups>
	<Minimum>0</Minimum>
	<Maximum>1</Maximum>

      "You can't have a sequence of exactly two night shifts, except
      at the ends":
      <ClusterBusyTimesConstraint Id="Constraint:12">
	<AppliesToTimeGroup Reference="1-3Wed-Tue1:4Wed-Mon1"/>
	<TimeGroups>
	  <TimeGroup Reference="1Wed4" Polarity="negative"/>
	  <TimeGroup Reference="1Thu4"/>
	  <TimeGroup Reference="1Fri4"/>
	  <TimeGroup Reference="1Sat4" Polarity="negative"/>
	</TimeGroups>
	<Minimum>0</Minimum>
	<Maximum>3</Maximum>

      "You can't have four night shifts in a row starting on a Monday":
      <LimitBusyTimesConstraint Id="Constraint:7">
        <AppliesToTimeGroup Reference="1-4Mon1"/>
	<TimeGroups>
	  <TimeGroup Reference="1Mon-Tue4:2Wed-Thu4"/>
	</TimeGroups>
	<Minimum>0</Minimum>
	<Maximum>3</Maximum>

      "You can't have four night shifts in a row starting on a Tuesday":
      <LimitBusyTimesConstraint Id="Constraint:1">
        <AppliesToTimeGroup Reference="1-3Tue1"/>
	<TimeGroups>
	  <TimeGroup Reference="1Tue4:2Wed-Fri4"/>
	</TimeGroups>
	<Minimum>0</Minimum>
	<Maximum>3</Maximum>

      "You can't have four night shifts in a row starting on a Thursday"
      <LimitBusyTimesConstraint Id="Constraint:2">
        <AppliesToTimeGroup Reference="1-4Thu1"/>
	<TimeGroups>
	  <TimeGroup Reference="1Thu-Sun4"/>
	</TimeGroups>
	<Minimum>0</Minimum>
	<Maximum>3</Maximum>

      "You can't have four night shifts in a row starting on a Friday":
      <LimitBusyTimesConstraint Id="Constraint:4">
        <AppliesToTimeGroup Reference="1-4Fri1"/>
	<TimeGroups>
	  <TimeGroup Reference="1Fri-Mon4"/>
	</TimeGroups>
	<Minimum>0</Minimum>
	<Maximum>3</Maximum>

      "You can't have four night shifts in a row starting on a Saturday":
      <LimitBusyTimesConstraint Id="Constraint:5">
        <AppliesToTimeGroup Reference="1-4Sat1"/>
	<TimeGroups>
	  <TimeGroup Reference="1Sat-Tue4"/>
	</TimeGroups>
	<Minimum>0</Minimum>
	<Maximum>3</Maximum>

      "You can't have four night shifts in a row starting on a Sunday":
      <LimitBusyTimesConstraint Id="Constraint:6">
        <AppliesToTimeGroup Reference="1-4Sun1"/>
	<TimeGroups>
	  <TimeGroup Reference="1Sun-Tue4:2Wed4"/>
	</TimeGroups>
	<Minimum>0</Minimum>
	<Maximum>3</Maximum>

    So this amounts to "you can't have four night shifts in a row
    unless the first night shift is on a Wednesday".

    Constraint:26 says "at least 10 days off" but vacation
    days count as days off, because if you are on vacation
    you are free at times 2-4.  Workloads are fairly tight
    but not at the limit.

    "Try to assign two days off after five consecutive shifts"

    "At least six days between two sets of night shifts".  Could
    this be converted to a limit active intervals constraint,
    and if so why isn't it being converted now?  Also there
    is no pattern for night - non-night - night, so it does
    actually allow one non-night between two sets of night shifts.

    Here's a weird one, labelled "At least two days off after night
    shift", max 2:

	<TimeGroups>
	  <TimeGroup Reference="1Wed4"/>
	  <TimeGroup Reference="1Thu2-4" Polarity="negative"/>
	  <TimeGroup Reference="1Fri2-4"/>
	</TimeGroups>

    But it pairs with this later on:

        <TimeGroups>
	  <TimeGroup Reference="1Wed4"/>
	  <TimeGroup Reference="1Thu2-3"/>
	</TimeGroups>
	<Maximum>1</Maximum>

    So you can't work a day shift after a night shift (this one),
    and (prev one) if you don't, then you can't work a day shift
    the day after that.  This is a hard constraint.

    Prefer morning shifts before a vacation period and a night shift
    afterwards - presumably correlated with the V preassignments.

    At least four Saturdays and/or Sundays - this means individual
    days, not whole weekends

    At least 10 days off - vacation days count as days off

    Avoid A followed by M (afternoon, morning)

    Avoid nights before requested days off - here requested days
    off mean Vacation days.

    (Min 6 / Max 8) Morning shifts
    (Min 5 / Max 7) Afternoon shifts
    (Max 7 / Min 5) Nights
    These seem to be moderately tight.

    Max four consecutive night shifts (weight 1)
    Min two consecutive night shifts (weight 1)

    Try to allocate days shifts in sets of three consecutive shifts
    This means morning+afternoon shifts, and it means maximum of 3.

    Max five consecutive working days (condensed) hard constraint -
    this means all non-vacation, non-free days.

    3:all, max 4 - max 4 staff on night shifts each day
    2:all, max 4 - max 4 staff on afternoons and weekend mornings
    1:all, max 5 - max 5 staff on weekday mornings
    3:all, max 5 - max 5 staff on all shifts
    3:all, min 4 - min 4 staff on night shifts

    We aren't merging 3:all, min 4 with 3:all, max 4, but that is
    because we are already doing a lot of other merging.

18 December 2018.  Added a repair that does double swaps during
  task augment.  First results:

    [ "COI-BCDT-Sep", 1 solution, in 29.0 secs: cost 0.00270 ]

    [ "COI-BCDT-Sep", 4 threads, 8 solves, 6 distinct costs, 69.1 secs:
      0.00230 0.00260 0.00260 0.00270 0.00280 0.00290 0.00290 0.00320
    ]

  This is opposed to

    [ "COI-BCDT-Sep", 1 solution, in 25.2 secs: cost 0.00340 ]

    [ "COI-BCDT-Sep", 4 threads, 8 solves, 6 distinct costs, 73.0 secs:
      0.00260 0.00310 0.00320 0.00320 0.00330 0.00340 0.00340 0.00350
    ]

  without these repairs.  So there is some improvement.  But more
  importantly there may be a more principled repair structure here.

  If we halve the number of double moves, from 12 to 6:

    [ "COI-BCDT-Sep", 4 threads, 8 solves, 6 distinct costs, 72.9 secs:
      0.00260 0.00310 0.00320 0.00320 0.00330 0.00340 0.00340 0.00350
    ]

  It seems to have made virtually no difference.  Let's try 3:

    [ "COI-BCDT-Sep", 4 threads, 8 solves, 6 distinct costs, 72.3 secs:
      0.00260 0.00310 0.00320 0.00320 0.00330 0.00340 0.00340 0.00350
    ]

  Seems like most of the repairs are not doing any good, but they
  may do good in other instances.  Let's try 24, for luck:

    [ "COI-BCDT-Sep", 4 threads, 8 solves, 6 distinct costs, 70.9 secs:
      0.00260 0.00310 0.00320 0.00320 0.00330 0.00340 0.00340 0.00350
    ]

  It actually ran faster.  Back to 12 I think:

    [ "COI-BCDT-Sep", 4 threads, 8 solves, 6 distinct costs, 66.5 secs:
      0.00230 0.00260 0.00260 0.00270 0.00280 0.00290 0.00290 0.00320
    ]

  Here is what happens when we insist that the tasks being swapped
  be not equivalent:

    [ "COI-BCDT-Sep", 4 threads, 8 solves, 6 distinct costs, 74.2 secs:
      0.00240 0.00250 0.00260 0.00260 0.00270 0.00290 0.00290 0.00330
    ]

  Somewhat worse, not sure why.

  But in any case KheTaskMoveAugment is the wrong place for
  this stuff, unless we add a "swaps OK" boolean flag.

  After bug fix,

    [ "COI-BCDT-Sep", 1 solution, in 27.4 secs: cost 0.00260 ]

    [ "COI-BCDT-Sep", 4 threads, 8 solves, 5 distinct costs, 73.2 secs:
      0.00230 0.00260 0.00260 0.00260 0.00270 0.00300 0.00300 0.00310
    ]

  Very little difference.

  Thinking again about classifying calls to augment functions,
  that is for event resource and resource constraints:

  * KheAssignResourceAugment, KhePreferResourcesAugment, and
    KheLimitResourcesAugment.  Here we want to change the
    assignment of one task so that it either lies within
    or does not lie within a particular resource group.

    We need to think again about which repairs are appropriate
    for this, and hopefully implement one or two functions that
    all of them can call, say KheEventGroupUnderload and
    KheEventGroupOverload.

  * KheAvoidUnavailableTimesAugment, KheClusterBusyTimesAugment,
    KheLimitBusyTimesAugment, KheLimitWorkloadAugment,
    KheLimitActiveIntervalsAugment.

    These all call KheUnderloadAugment and/or KheOverloadAugment;
    they all work by increasing or decreasing (possibly to zero)
    the number of busy times for some resource in some time group.

  So at the top level we have the top-level augment functions, at
  the next level we have KheUnderloadAugment, KheOverloadAugment,
  and the new KheEventGroupUnderload and KheEventGroupOverload.
  Below those we just call whichever repairs seem best.

  We now think of KheTaskMoveAugment and KheTaskSetMoveAugment
  as moving a task or task set away from a resource, leaving the
  resource unassigned afterwards where the task or task set was.
  This is suited to KheOverloadAugment *only*.  The other calls
  on these two functions (from KheUnderloadAugment,
  KheAssignResourceAugment, KhePreferResourcesAugment,
  and KheLimitResourcesAugment) need a rethink.

19 December 2018.  Working on revising pap_solve to express my current
  ideas about the operations.  It's pretty well all done.

20 December 2018.  Audited the revised pap_solve.  It's ready to
  implement.

    KheAssignResourceAugment
      Monitors a single event resource, resources are all, all
      defects are underloads
  
    KhePreferResourcesAugment
      Monitors a single event resource, resources are a given set,
      all defects are underloads

    KheLimitResourcesAugment
      Monitors a set of event resources, resources are a given set,
      defects can be underloads or overloads

  Underload:  task must be assigned a resource r1 from outside the
  desired set, or NULL.  Assign a resource r2 from inside the
  desired set.  When reflecting, r2's current task must not be
  from inside the monitored set.

  Overload:  task must be assigned a resource r1 from inside the
  desired set.  Assign a resource r2 from outside the desired set,
  or NULL.  When reflecting, r2's current task must not be from
  inside the monitored set.

  KheResourceUnderloadAugment(r, tg, allow_zero):

  KheResourceOverloadAugment(r, tg, require_zero);

  KheEventResourceUnderload(er, d):

  KheEventResourceOverload(er, d):

21 December 2018.  Thinking about new constraint names:

  ASSIGN_TIME			MEET_ASSIGNMENT_CONSTRAINT
  PREFER_TIMES
  LIMIT_TIMES (?)

  SPLIT_EVENTS			MEET_SPLIT_CONSTRAINT
  DISTRIBUTE_SPLIT_EVENTS

  SPREAD_EVENTS			MEET_SPREAD_CONSTRAINT
  LINK_EVENTS
  ORDER_EVENTS

  ASSIGN_RESOURCE		TASK_ASSIGNMENT_CONSTRAINT
  PREFER_RESOURCES
  LIMIT_RESOURCES

  AVOID_SPLIT_ASSIGNMENTS	TASK_STABILITY_CONSTRAINT

  AVOID_CLASHES	(> 1)		RESOURCE_TOTAL_CONSTRAINT
  AVOID_UNAVAILABLE_TIMES         A set of sets of times.  For
  CLUSTER_BUSY_TIMES (> 0)        each set, a rule for whether
  LIMIT_BUSY_TIMES                the set is active, which could
  LIMIT_WORKLOAD                  be >1 for clashes, >0 for positive,
                                  and <=0 for negative.
				  <bound type="workload" dir="neg">1</bound>

  LIMIT_ACTIVE_INTERVALS	RESOURCE_CONSECUTIVE_CONSTRAINT
  LIMIT_IDLE_TIMES

  Tidying up event resource augments.  They now work by calling
  KheEventResourceUnderloadAugment and KheEventResourceUnderloadAugment.

22 December 2018.  Continuing to implement the new design.  Wrote
  KheResourceTimetableMonitorTaskSetBusy, have to use it now.

23 December 2018.  Documented KheResourceTimetableMonitorTaskSetBusy.
  Finished writing the new version, it needs a careful audit now.

24 December 2018.  Audited and revised KheResourceUnderloadAugment.
  Started testing the new code.

25 December 2018.  Trying to avoid preassigned tasks in a principled way.
  All done and ready to test.  First results on COI-BCDT-Sep:

    [ "COI-BCDT-Sep", 1 solution, in 29.7 secs: cost 0.00250 ]

    [ "COI-BCDT-Sep", 4 threads, 8 solves, 5 distinct costs, 71.5 secs:
      0.00220 0.00230 0.00230 0.00230 0.00230 0.00240 0.00250 0.00280
    ]

  These are easily my best results so far on this instance, although
  the running time has blown out.  The optimal solution has cost 100.

  Did a full run of COI.  The results are best so far on cost,
  running time a little slow but perfectly acceptable.  The only
  really poor result is COI-Valouxis-1.  I have to look into that.

  KheEventResourceOverloadAugment and KheEventResourceUnderloadAugment
  now replaced by KheEventResourceMoveAugment.  First results:

    [ "COI-Valouxis-1", 1 solution, in 1.2 secs: cost 0.00540 ]

    [ "COI-Valouxis-1", 4 threads, 8 solves, 7 distinct costs, 2.7 secs:
      0.00120 0.00260 0.00300 0.00360 0.00360 0.00460 0.00540 0.01420
    ]

  This 120 is nearly optimal, it's my best ever for COI-Valouxis-1.
  It would be good though to get more consistency.  But all the
  defects are "3 length stretches", i.e. not exactly 4 in a row.
  So let's try a full COI run again.

  Did the full COI run, there are now 11 optimal results which is
  the best so far I believe.  And generally the results are very
  good indeed.  It might be worth looking at, say, CHILD in detail.

  Done an INRC1 run.  Average cost is the same, run time is
  slower than it was earlier today.

  Started a CQ14 run.  It finished safely, I haven't checked
  the results yet.  They are in khe18.pdf.

26 December 2018.  Looked over yesterday's CQ14 run.  It is in
  khe18-12-26.pdf.  Best results so far for CQ14, clearly better
  now than CQ-EJ10.

  Wrote this some time ago:  "Can I see what defects a repair
  produces, and tune subsequent repairs accordingly?  I could
  use this to decide whether to go on and do double day-set
  swaps after an ordinary day-set swap."  It's not a bad idea,
  but it seems to be hard to carry out.  I've more or less
  abandoned it.

  Wrote this some time ago:  "If the cost function is a step
  function, only repairs which reduce the amount of deviation
  to zero are worth trying.  This is not part of the current
  thinking.  I've documented this in the solve paper but not
  done anything about it."  I've more or less decided to do
  nothing about this, because it is rare, and I can't test it.

  I'm approaching a point now where it is not likely that I
  will be able to reduce costs much further.  But it would
  be good to reduce run times if possible.

  Off-site backup done today.

  COI-BCDT-Sep before I started fiddling:

    [ "COI-BCDT-Sep", 1 solution, in 32.2 secs: cost 0.00250 ]

  After modifying KheTaskMoveAugment to visit all equivalent
  tasks when the task is initially unassigned:

    [ "COI-BCDT-Sep", 1 solution, in 30.8 secs: cost 0.00210 ]

    [ "COI-BCDT-Sep", 4 threads, 8 solves, 4 distinct costs, 67.0 secs:
      0.00200 0.00200 0.00200 0.00200 0.00210 0.00240 0.00270 0.00270
    ]

  Marginally faster, and better solutions.

  Had a good hunt through the COI-BCDT-Sep defects, it is very
  hard to see how to make further progress.  As evidence for
  this I've tried KHE18x32, results are

    [ "COI-BCDT-Sep", 4 threads, 32 solves, 8 distinct costs, 245.2 secs:
      0.00180 0.00180 0.00190 0.00190 0.00200 0.00200 0.00200 0.00210
      0.00210 0.00210 0.00210 0.00210 0.00210 0.00220 0.00220 0.00220
      0.00220 0.00220 0.00220 0.00220 0.00220 0.00220 0.00220 0.00220
      0.00240 0.00240 0.00240 0.00260 0.00260 0.00270 0.00270 0.00270
    ]

27 December 2018.  I've decided to start work on the detailed testing,
  that is, deleting one feature of the algorithm at a time and seeing
  what effect it has.  It might be that time is being wasted on things
  that do no good; if so, we need to know that now, because the main
  remaining problem is run time.

  I've added these labels to the paper:

    R2	Run the repair phase twice
    CE	Ejection chain repair during construction
    CW  Edge cost adjustment to take account of resource workload
    CL	Edge cost adjustment to take account of constraint limits
    CB	Edge cost adjustment to minimize consecutive busy days
    CS	Avoid spurious costs
    CR	Rematching during constraints
    CI  Individual day rematching at the end of construction
    RM	Rematching during the repair phases
    EE	Ejection chain repair during the repair phases
    EW	Widening during ejection chain repair
    ER	Reversing during ejection chain repair
    EB	Balancing during ejection chain repair
    GR	Grouping event resources

  Including cost and time there is only room for three variants
  (not counting KHE18) per table.  A possible arrangement is

    General
    -------
    GR	Grouping event resources
    CS	Avoid spurious costs

    Construction
    ------------
    CE	Ejection chain repair during construction
    CR	Rematching during construction
    CI  Individual day rematching at the end of construction

    Edge cost adjustments during time sweep
    ---------------------------------------
    CW  Edge cost adjustment to take account of resource workload
    CL	Edge cost adjustment to take account of constraint limits
    CB	Edge cost adjustment to minimize consecutive busy days

    Repair
    ------
    R2	Run the repair phase twice
    RM	Try rematching during the repair phases
    EE	Try ejection chain repair during the repair phases

    Ejection chain repair
    ---------------------
    EW	Widening during ejection chain repair  (es_widening_off)
    ER	Reversing during ejection chain repair  (es_reversing_off)
    EB	Balancing during ejection chain repair  (es_balancing_off)

  I've decided that it has to be KHE18x8 (worse luck).  The
  results so far are too erratic.  Run is finished.

28 December 2018.  Looked over yesterday's results, and I've
  now done all five tables.  They say that they are for KHE18x8
  but at present the last three are just for KHE18.  Have to fix
  that but only at a time when the long runs will not hold me up.

29 December 2018.  Finished a complete run of all tables, including
  the new aspects tables all run with KHE18x8.  Now in khe18-12-29.pdf.

  Started work on INRC2-4-035-2-8875.xml, discovered that the
  constraint requiring at most one shift per day was missing.
  Found and fixed the bug in nrconv/inrc2.c that was causing
  this.  Previously a basic run produced

    [ "INRC2-4-035-2-8875", 1 solution, in 13.4 secs: cost 0.01700 ]

  Now the same run is producing

    [ "INRC2-4-035-2-8875", 1 solution, in 46.7 secs: cost 0.01965 ]

  It's not very encouraging, but I need to do a full run now to
  see where I am.  Most of the algorithms do not introduce clashes
  anyway which is why my previous solutions were not totally off.

30 December 2018.  Put the new INRC2 archive files on the web site.
  Curiously there did not seem to be any old versions there, even
  though index.html had links to them.

  TaskSetDoubleMove was swapping equivalent tasks, which is a waste
  of time.  After removing that I got

    [ "INRC2-4-035-2-8875", 1 solution, in 59.2 secs: cost 0.01925 ]

  which is even slower but a slightly better cost.  Tried doing the
  free resources before the busy ones, but got this:

    [ "INRC2-4-035-2-8875", 1 solution, in 63.3 secs: cost 0.01940 ]

  so I've abandoned that idea.

  Had a look at the grouping and found that too much was going on.
  I've corrected the problem that caused that, and now I get

    [ "INRC2-4-035-2-8875", 1 solution, in 21.1 secs: cost 0.01650 ]

  Jeremy Omer's solution has cost 1155, so we've made a big step
  in the right direction here, both in cost and in time.  Best of 8:

    [ "INRC2-4-035-2-8875", 4 threads, 8 solves, 8 distinct costs, 57.2 secs:
      0.01565 0.01585 0.01615 0.01650 0.01660 0.01690 0.01705 0.01815
    ]

  Added handling of the allow_zero case to the cluster augment function.
  At present it only does anything when there is exactly one active time
  group, but that will be enought to fix some complete weekends problems.
  It seems to be helping:

    [ "INRC2-4-035-2-8875", 1 solution, in 28.9 secs: cost 0.01610 ]

    [ "INRC2-4-035-2-8875", 4 threads, 8 solves, 8 distinct costs, 58.5 secs:
      0.01550 0.01605 0.01610 0.01615 0.01695 0.01705 0.01740 0.01820
    ]

  There is still one complete weekends defect, but it is right at the
  end of the cycle and seems difficult.

  Did a full COI run.  The average cost has improved, although it
  may be just noise.  The running time is a bit better too, and
  consistently so, so that is not noise.

  I found that available workload was not being calculated when the
  relevant cluster busy times constraint had one time group for each
  time.  Before the fix the unrepaired cost was 2170, after the fix
  it was 2220, which is not encouraging.  But back to a full run:

    [ "INRC2-4-035-2-8875", 1 solution, in 14.8 secs: cost 0.01820 ]

    [ "INRC2-4-035-2-8875", 4 threads, 8 solves, 8 distinct costs, 52.7 secs:
      0.01585 0.01635 0.01690 0.01700 0.01760 0.01800 0.01820 0.01835
    ]

  So no significant change (it's a bit worse), although it is running
  somewhat faster, and the two full-time nurses (NU_8 and NU_9) are
  better utilized, both are at 2 rather than 3 and 5 as before.

  TR_30 has a break on day 2 because of history in Constraint:20.

  CT26 -> CT20 on 4Thu.  Should improve things, why not done?
  Lots of reasons.

31 December 2018.  Designed, implemented, documented, and tested
  resource swapping.  Without swapping:

    [ "INRC2-4-035-2-8875", 1 solution, in 26.3 secs: cost 0.01585 ]

    [ "INRC2-4-035-2-8875", 4 threads, 8 solves, 8 distinct costs, 47.7 secs:
      0.01585 0.01635 0.01690 0.01700 0.01760 0.01800 0.01820 0.01835
    ]

  With swapping:

    [ "INRC2-4-035-2-8875", 1 solution, in 32.9 secs: cost 0.01550 ]

    [ "INRC2-4-035-2-8875", 4 threads, 8 solves, 8 distinct costs, 51.2 secs:
      0.01550 0.01655 0.01690 0.01700 0.01760 0.01800 0.01820 0.01835
    ]

  So it does help, but it's a bit slow.  I'll keep it on for now.
  After all it is quite different from all the other repairs.

  Added light grey boxes to timetables, denoting soft unavailable days.

  Implemented edge_adjust4 in resource matching, which favours
  assigning the same shift type on successive days.  First results:

    [ "INRC2-4-035-2-8875", 1 solution, in 36.4 secs: cost 0.01630 ]

    [ "INRC2-4-035-2-8875", 4 threads, 8 solves, 8 distinct costs, 69.0 secs:
      0.01565 0.01630 0.01645 0.01655 0.01670 0.01760 0.01800 0.01895
    ]

  It's slower and the results are worse.  But must look at them in
  detail first, to see why.  Turning of adjust3 and adjust4 when
  the time set is not immediately after the previous one gives

    [ "INRC2-4-035-2-8875", 1 solution, in 34.0 secs: cost 0.01700 ]

    [ "INRC2-4-035-2-8875", 4 threads, 8 solves, 8 distinct costs, 65.1 secs:
      0.01635 0.01670 0.01700 0.01715 0.01740 0.01750 0.01765 0.01775
    ]

  Turning off edge adjust 3 (which tries for short sequences):

    [ "INRC2-4-035-2-8875", 1 solution, in 27.4 secs: cost 0.01680 ]

    [ "INRC2-4-035-2-8875", 4 threads, 8 solves, 8 distinct costs, 64.8 secs:
      0.01635 0.01670 0.01700 0.01715 0.01740 0.01750 0.01765 0.01775
    ]

  This is all bumping along the bottom.  I need some better ideas.


To Do - now carried forward to 2019
===================================

  The last few ideas have been too small beer.  Must do better!

  I've tried to encourage time sweep to assign the same shift type to
  the same resource on successive days.  This seems to make things
  slightly worse, I need to work out why.

  What about grouping on the fly?  If we are about to assign one
  day of a time sweep, we can deduce that an ungrouped night shift
  should be grouped with at least three others in the next asst.

  Currently trying to improve KHE18's solutions to INRC2-4-035-2-8875.xml:

    1 = Early, 2 = Day, 3 = Late, 4 = Night
    FullTime: max 2 weekends, 15-22 shifts, consec 2-3 free 3-5 busy
    PartTime: max 2 weekends,  7-15 shifts, consec 2-5 free 3-5 busy
    HalfTime: max 1 weekends,  5-11 shifts, consec 3-5 free 3-5 busy
    All: unwanted [4][123], [3][12], complete weekends, single asst per day
    All: consec same shift days: Early 2-5, Day 2-28, Late 2-5, Night 4-5

    Resource types

    HeadNurse (HN_*) can also work as Nurse or Caretaker
    Nurse     (NU_*) can also work as Caretaker
    Caretaker (CT_*) works only as Caretaker
    Trainee   (TR_*) works only as Trainee

  "At least two days off after night shift" - if we recode
  this, we might do better on COI-BCDT-Sep.

  Option es_fresh_visits seems to be inconsistent, it causes
  things to become unvisited when there is an assumption that
  they are visited.  Needs looking into.  Currently commented
  out in khe_sr_combined.c.

  Special issue deadline is now 28 February 2019.

  YIKES - I submitted the wrong version of the modelling
  paper; conf1 rather than journal1

  Install new version of HSEval to fix character set problem.

  For the future:  time limit storing.  khe_sm_timer.c already
  has code for writing time limits, but not yet for reading.

  I've now confirmed that HSEval's evaluations of the INRC2
  solutions agree with the validator.txt evaluations in all
  cases - all of Omer's evaluations, and the one solution that
  Andrea sent.  Need the rest of Andrea's solutions now.  I've
  emailed him asking for them.  Time has passed and he has
  not responded.  At some point towards the end of October
  I will have to cut him out.

  Work on time modelling paper for PATAT 2020.  The time model
  is an enabler for any projects I might do around ITC 2019,
  for example modelling student sectioning and implementing
  single student timetabling, so it is important for the future
  and needs to be got right.

  Add INRC2 instances to the XESTT web site - the four week
  ones from Andrea's PATAT 2018 paper.  I have now received
  solutions from Jeremy and Andrea.

  Time sets, time groups, resource sets, and resource groups
  ----------------------------------------------------------

    Thinking about whether I can remove construction of time
    neighbourhoods, by instead offering offset parameters on
    the time set operations (subset, etc.) which do the same.

    Need to use resource sets and time sets a lot more in the
    instance, for the constructed resource and time sets which
    in general have no name.  Maybe replace solution time groups
    and solution resource groups altogether.  But it's not
    trivial, because solution time groups are used by meets,
    and solution resource groups are used by tasks, both for
    handling domains (meet and task bounds).  What about

      typedef struct khe_time_set_rec {
          SSET elems;
      } KHE_TIME_SET;

    with SSET optimized by setting length to -1 to finalize.
    Most of the operations would have to be macros which
    add address-of operators in the style of SSET itself.

       KHE_TIME_SET KheTimeSetNeighbour(KHE_TIME_SET ts, int offset);

    would be doable with no memory allocation and one binary
    search (which could be optional for an internal version).

    I'm leaving this lie for now, something has to be done
    here but I'm not sure what, and there is no great hurry.

  There is a problem with preparing once and solving many times:
  adjustments for limit resources monitors depend on assignments
  in the vicinity, which may vary from one call to another.  The
  solution may well be simply to document the issue.

  At present resource matching is grouping then ungrouping during
  preparation, then grouping again when we start solving.  Can this
  be simplified?  There is a mark in the way.

  Document sset (which should really be khe_sset) and khe_set.

  What to do about tasks without assign resource or limit
  resources monitors?  What about including them in the
  matching but with an edge adjustment that encourages
  non-assignment?  This needs looking into in general.
  At present I am omitting these tasks, because I get
  better results on GPost when I do; but I need to think
  about this and do some serious experiments.

  I'm slightly worried that the comparison function for NRC
  worker constraints might have lost its transitivity now that
  history_after is being compared in some cases but not others.

  Look at the remaining special cases in all.map and see if some
  form of condensing can be applied to them.

  Develop my ideas for a generalized version of XHSTT/XESTT,
  but low priority.  Maybe something for PATAT 2020, when
  I've looked at the university course timetabling instances
  from the forthcoming competition.

  Might be a good idea to review the preserve_existing option in
  resource matching.  I don't exactly understand it at the moment.

  Option es_active_augment is currently undocumented.  Eventually
  I need to either remove it or document it.  At present, best of
  8 on COI-Post gives cost 10 with es_active_augment=swap, and
  cost 35 with es_active_augment=move, so I have made swap be
  its default value.

  There seem to be several silly things in the current code that are
  about statistics.  I should think about collecting statistics in
  general, and implement something.  But not this time around.

  KheTaskFirstUnFixed is quite widely used, but I am beginning to
  suspect that KheTaskProperRoot is what is really wanted.  I need
  to analyse this and perhaps make some conceptual changes.

  Read the full GOAL paper.  Are there other papers whose aims
  are the same as mine (which GOAL's are not)?  If so I need
  to compare my results with theirs.  The paper is in the 2012
  PATAT proceedings, page 254.  Also it gives this site:

    https://www.kuleuven-kulak.be/nrpcompetition/competitor-ranking

  Can I find the results from the competition winner?  According to
  Santos et al. this was Valouxis et al, but their paper is in EJOR.

  Add code for limit resources monitors to khe_se_secondary.c.

  In KheClusterBusyTimesAugment, no use is being made of the
  allow_zero option at the moment.  Need to do this some time.

  Generalize the handling of the require_zero parameter of
  KheOverloadAugment, by allowing an ejection tree repair
  when the ejector depth is 1.  There is something like
  this already in KheClusterOverloadAugment, so look at
  that before doing anything else.

  There is an "Augment functions" section of the ejection chains
  chapter of the KHE guide that will need an update - do it last.

  (KHE) What about a general audit of how monitors report what
  is defective, with a view to finding a general rule for how
  to do this, and unifying all the monitors under that rule?
  The rule could be to store reported_deviation, renaming it
  to deviation, and to calculate a delta on that and have a
  function which applies the delta.  Have to look through all
  the monitors to see how that is likely to pan out.  But the
  general idea of a delta on the deviation does seem to be
  right, given that we want evaluation to be incremental.

  (KHE) For all monitors, should I include attached and unattached
  in the deviation function, so that attachment and unattachment
  are just like any other update functions?

  Ejection chains idea:  include main loop defect ejection trees
  in the major schedule, so that, at the end when main loop defects
  have resisted all previous attempts to repair them, we can try
  ejection trees on each in turn.  Make one change, produce several
  defects, and try to repair them all.  A good last resort?

  Ejection chains idea:  instead of requiring an ejection chain
  to improve the solution by at least (0, 1), require it to
  improve it by a larger amount, at first.  This will run much
  faster and will avoid trying to fix tiny problems until there
  is nothing better to do.  But have I already tried it?  It
  sounds a lot like es_limit_defects.
