KHE diary for 2013
------------------

2013-2014 work on KHE, solving XHSTT-2013.xml for PATAT2014

11 Oct 2013.  Began work today.  I need a road map of how all the
  solvers work together, and an audit to see whether they are doing
  everything efficiently.  At present the instances with individual
  students (3 Danish, 2 Dutch) have a large number of layers and are
  running far too slowly.

  Call graph of time solvers
  --------------------------

  main (all boilerplate)
    KheArchiveParallelSolve
    KheTestSolveInstances
      KheArchiveParallelSolve
      KheGeneralSolve
  KheGeneralSolve
    KheCycleNodeAssignTimes

  KheCycleNodeAssignTimes (mostly boilerplate, also runarounds)
    KheCoordinateLayers
      This function finds cases where pairs of resources are similar
      and makes one resource's nodes children of the other resource's.
    KheBuildRunarounds
      This function builds runarounds
    KheNodeSimpleAssignTimes
      This function unassigns all times in the children of the node
      it is given, then reassigns them in decreasing duration order,
      making the best assignment it can find in the current state
    KheNodeRecursiveAssignTimes
      This function applies a given one-level time solver to all
      descendants of the node it is given, in postorder
    KheRunaroundNodeAssignTimes
      This function assigns times to runaround nodes.  It is passed
      to KheNodeRecursiveAssignTimes so that this is done recursively.
    KheNodePreassignedAssignTimes
      Carry out assignments of preassigned times
    KheNodeLayeredAssignTimes
    KheEjectionChainRepairTimes
      We know what this does

  KheNodeLayeredAssignTimes
    KheLayerParallelAssignTimes
      Propagate preassignments in one layer to similar layers
    KheLayerMatchAssignTimes
      Use a weighted matching to heuristically assign one layer
    KheEjectionChainRepairTimes
      We know what this does

12 Oct 2013.  Compared khe.h with doc, all in order now.  Have
  also reorganized the makefile and sorted out new file names
  for structural solvers and time solvers, as well as a new
  distribution of functions into the newly named files.

13 Oct 2013.  Finished sorting out new file names and reorganizing.
  Also changed the layer tree construction file names, type names,
  and function names to replace "task" by "job".

14 Oct 2013.  Wrote a function which explodes one archive into one
  archive for each instance in the original archive.  So now I can
  work directly on DK-FG-12.xml and find out why it is so slow.

  Modified KheNodeLayeredAssignTimes so that if a layer is already
  completely assigned, it is skipped completely.  Even a repair is
  no use, because the repair does not know which layer we are up to
  anyway, so it can only make progress if something has changed
  since the last layer, i.e. if something has been assigned.

  Documented, implemented, and tested a modification to layer tree
  bulding which merges nodes whose meets are clearly from the same
  original course, pre-split by the instance.  Should make the tree
  smaller and easier to comprehend.  As part of this, documented and
  implemented KheEventMergeable and KheEventSharePreassignedResource.

15 Oct 2013.  Designed, documented, implemented, and tested an
  exponential backoff module.  But the terminology could be better.

16 Oct 2013.  Finalized the exponential backoff module.  Changed
  time repair solvers and their documentation so that they return
  true if they have been useful.  Used exponential backoff to
  control the calls to ejection repair after each layer.  But it
  did not decline may opportunities:

    82 opportunities (54 successful, 14 failed, 14 declined)

  There are too many opportunities, rather than too many
  unsuccessful ones, apparently.

  Documented a new version of ejectors which allows the main
  loop to be entered several times, with a different schedule
  each time.  Implementing and using it are next.

17 Oct 2013.  Implemented the new version of ejectors, and now
  have a default mode which explores everything at depth 1, then
  depth 2, then depth 3, then unlimited.  I've tested it and it
  seems to be working; the successes seem to be at increasing
  depth now.

  Discovered that dropping the vizier node speeds things up, to

      cost 0.0623 in 117.25 secs (that is, 2 minutes)
      [ Soln (instance "DK-FG-12", diversifier 0, cost 0.0623)
	Soln                              Defects          Cost
	-------------------------------------------------------
	SpreadEventsMonitor                    62        0.0059
	LimitIdleTimesMonitor                 260        0.0380
	ClusterBusyTimesMonitor               119        0.0073
	LimitBusyTimesMonitor                  88        0.0111
	-------------------------------------------------------
	Total                                 529        0.0623
      ]

  Then switching to repairing just the current layer produced

      cost 0.0730 in 29.34 secs
      [ Soln (instance "DK-FG-12", diversifier 0, cost 0.0730)
	Soln                              Defects          Cost
	-------------------------------------------------------
	SpreadEventsMonitor                    67        0.0058
	LimitIdleTimesMonitor                 302        0.0482
	ClusterBusyTimesMonitor               127        0.0074
	LimitBusyTimesMonitor                  87        0.0115
	-------------------------------------------------------
	Total                                 583        0.0730
      ]

  These results are fairly comparable, given that we're not
  repairing several of these constraint types at the moment.

  Now displaying 5 decimal places of soft cost everywhere and always;
  I've removed the option of displaying any other number of places.

18 Oct 2013.  It makes sense to detach limit idle times monitors
  during time assignment, since they report gaps which may be filled
  later as times are assigned to events.  But it's no big deal.

19 Oct 2013.  I've written up an abbreviated description of the
  node swap and Kempe meet move operations, for inclusion in the
  new paper, preceded by a general discussion of repair operations
  which is an attempt to clear my own head on the subject.  I need
  to add resource repair operations to this discussion next.

20 Oct 2013.  Worked on the proto-paper today, including a short
  description of how to repair the various kinds of defects,
  except not cluster busy times defects yet; they seem to be hard.

21 Oct 2013.  Worked on the paper today, it's going well.

23 Oct 2013.  Still working on the paper.  I've more or less done
  about as much as I can with it for now; it's time to audit the code
  and align it with the description, including adding new stuff.

31 Oct 2013.  Had a week away, returned to working on the paper today.
  Went through it carefully.

1 Nov 2013.  Working on coordinating the descriptions in the paper
  with the actual code.  Currently I'm working on the structural
  phase, specifically KheLayerTreeMake.  I may need to go for a
  more radical rewrite, one which follows the code more closely.

2 Nov 2013.  Finished a description of the structural phase which
  is good enough to be going on with.  I should now review all the
  code for the structural and time assignment phases in detail.

  Decided to add Fix and UnFix operations to the basic aspects.
  All documented, and I've implemented the operations and the
  prevention of changes while fixed, but all the consequences
  wrt attaching and detaching monitors are still to do.

3 Nov 2013.  Working on Fix and UnFix.  All done for meet splits
  and meet domains, some work done on meet assigns (but needs more
  thought), revised the spec and implementation of KheMeetLeader,
  and have begun work on task domains.  Also have to think about
  KheTaskLeader.  May be best left as is, this time around anyway.

4 Nov 2013.  Finished Fix and UnFix for task domains and assignments.
  And I finally found the right way to document fixing and unfixing.

5 Nov 2013.  Audited KheMeetSplitFix and KheMeetSplitUnFix,
  KheMeetDomainFix and KheMeetDomainUnFix, KheTaskDomainFix and
  KheTaskDomainUnFix, KheTaskAssignFix and KheTaskAssignUnFix.
  Just KheMeetAssignFix and KheMeetAssignUnFix to do now, with
  consequences for link events and order events constraints.

  Implemented KheMeetAssignFix and KheMeetAssignUnFix, but the
  consequent calls on link events and order events monitors
  are stubs at the moment.  Have also removed KheMeetLeader
  and begun work on updating KheLayerTreeMake (done the docs).

  KheMergeMeets is next - we're going through all calls that
  change fixable things and making sure they are unfixed.
  Have removed all trace of AttachCheck, leader meets, and
  leader tasks from the documentation; still have to remove
  them from the implementation.

6 Nov 2013.  More work on the documentation, making the
  description of tasks more like the description of meets.
  Have decided to have leader meets and tasks by another
  name:  KheMeetFirstUnFixed and KheTaskFirstUnFixed.
  Now have clean compile.

  Gave KheMeetUnAssign and KheMeetUnAssignTime a bool result
  and added KheMeetUnAssignCheck and KheMeetUnAssignTimeCheck.
  Audited all uses of KheMeetUnAssign to ensure they handle
  the false return case, when that is possible.  Ditto for
  KheTaskUnAssign and KheTaskUnAssignResource.

  Finished Fix and Unfix for link events and order events monitors.

7 Nov 2013.  Did some testing, got a final solution for BGHS98,
  including quite a lot of debugging.

  Added task domain fixing outside cycle tasks.  Added the 10 Fix
  and UnFix operations to the list of operations that are handled
  by transactions.  We really need this.

8 Nov 2013.  Compared solver code with what I've written.  All
  in good shape now.  Got domain tightening working (had to fix
  one horrible bug in new code - domain fixing in transactions).
  Here are some tests:

  Ejector (1,revisit)+(inf,norevisit) and no two-resource:

    KheGeneralSolve2014 at end (12.97 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 9.00532)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   3       7.00000
      AvoidSplitAssignmentsMonitor           38       0.00460
      SpreadEventsMonitor                     8       0.00008
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  27       0.00064
      -------------------------------------------------------
      Total                                  78       9.00532
    ]

  Ejector (12,revisit)+(inf,norevisit) and no two-resource:

    KheGeneralSolve2014 at end (16.49 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 7.00544)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   2       5.00000
      AvoidSplitAssignmentsMonitor           39       0.00450
      SpreadEventsMonitor                    28       0.00028
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  27       0.00066
      -------------------------------------------------------
      Total                                  98       7.00544
    ]

  Ejector (123,revisit)+(inf,norevisit) and no two-resource:

    KheGeneralSolve2014 at end (61.24 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 6.00532)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   1       4.00000
      AvoidSplitAssignmentsMonitor           35       0.00440
      SpreadEventsMonitor                    32       0.00032
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  24       0.00060
      -------------------------------------------------------
      Total                                  94       6.00532
    ]

  Altogether the longer ejector runs are paying off, but too
  slow for daily use, so going with (1,revisit)+(inf,norevisit).
  Now (1,revisit)+(inf,norevisit) and two-resource:

    KheGeneralSolve2014 at end (13.40 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 9.00385)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   3       7.00000
      AvoidSplitAssignmentsMonitor           30       0.00320
      SpreadEventsMonitor                     7       0.00007
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  25       0.00058
      -------------------------------------------------------
      Total                                  67       9.00385
    ]

  There is a cost improvement here, and only 0.43 seconds slower.
  Let's stick with this for now.  Moving the second ejection
  chain repair to after two-resource is just a bit worse:

    KheGeneralSolve2014 at end (12.31 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 9.00449)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   3       7.00000
      AvoidSplitAssignmentsMonitor           31       0.00380
      SpreadEventsMonitor                     7       0.00007
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  27       0.00062
      -------------------------------------------------------
      Total                                  70       9.00449
    ]

  so we won't do that.  Repairing both before and after gives

    KheGeneralSolve2014 at end (14.04 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 9.00383)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   3       7.00000
      AvoidSplitAssignmentsMonitor           30       0.00320
      SpreadEventsMonitor                     7       0.00007
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  25       0.00056
      -------------------------------------------------------
      Total                                  67       9.00383
    ]

  which is not significantly better either, and fairly slow.

9 Nov 2013.  Wrote up a fairly careful description of the
  weighted matching resource repair algorithm I used once
  before.  The next step is to re-implement it in KHE.  I've
  started on it; I am now generating the correct sets of times.

10 Nov 2013.  Implemented resource rematching, which found two
  successes: 1369.00557 -> 1369.00547 and 1369.00547 -> 1369.00527.
  Here is the final result with resource rematching:

    KheGeneralSolve2014 at end (16.83 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 9.00414)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   3       7.00000
      AvoidSplitAssignmentsMonitor           32       0.00350
      SpreadEventsMonitor                     6       0.00006
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  25       0.00058
      -------------------------------------------------------
      Total                                  68       9.00414
    ]

  and here it is without resource rematching:

    KheGeneralSolve2014 at end (14.08 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 9.00385)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   3       7.00000
      AvoidSplitAssignmentsMonitor           30       0.00320
      SpreadEventsMonitor                     7       0.00007
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  25       0.00058
      -------------------------------------------------------
      Total                                  67       9.00385
    ]

  So we are actually better off without it at the moment.  In
  fact the real cause of these problems is that there are 6
  unmatchable demand tixels at the end of time assignment:

    KheGeneralSolve2014 after time assignment (6.76 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 2793.00033)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                 666    2787.00000
      SpreadEventsMonitor                    33       0.00033
      OrdinaryDemandMonitor                   6       6.00000
      -------------------------------------------------------
      Total                                 705    2793.00033
    ]
  
  If we can't get rid of them during time assignment, we can't
  hope to get rid of them later.  We need better time assignment.

11 Nov 2013.  Obtained 8 diversified solutions for AU-BG-98.
  Costs were 4.00512, 9.00385, 9.00385, 9.00418, 9.00458,
  10.00444, 10.00444, and 12.00547.  Which proves that there
  is a better solution out there, if only we can find it.

  Got an improvement from 4.00512 to 3.00500 by sorting event
  resource defects from before copying them:

    KheGeneralSolve2014 at end (12.10 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 4, cost 3.00500)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   1       2.00000
      AvoidSplitAssignmentsMonitor           42       0.00460
      AvoidClashesMonitor                     1       1.00000
      LimitBusyTimesMonitor                  16       0.00040
      -------------------------------------------------------
      Total                                  60       3.00500
    ]

  Then after sorting defects before copying them in all cases
  where I copy them at all I got this:

    [ Soln (instance "AU-BG-98", diversifier 4, cost 3.00486)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   1       2.00000
      AvoidSplitAssignmentsMonitor           38       0.00410
      SpreadEventsMonitor                    24       0.00024
      AvoidClashesMonitor                     1       1.00000
      LimitBusyTimesMonitor                  24       0.00052
      -------------------------------------------------------
      Total                                  88       3.00486
    ]

  so it looks like sorting before copying is here to stay.
  Without setting the diversifier I now get

    KheGeneralSolve2014 at end (14.59 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 9.00407)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   3       7.00000
      AvoidSplitAssignmentsMonitor           31       0.00350
      SpreadEventsMonitor                     5       0.00005
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  21       0.00052
      -------------------------------------------------------
      Total                                  62       9.00407
    ]

  which is marginally worse (and slower) than the 9.00385 before:

    KheGeneralSolve2014 at end (13.40 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 9.00385)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   3       7.00000
      AvoidSplitAssignmentsMonitor           30       0.00320
      SpreadEventsMonitor                     7       0.00007
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  25       0.00058
      -------------------------------------------------------
      Total                                  67       9.00385
    ]

  but still I think we'll stick with sorting.  The *number* of
  defects is lower, and a hard cost of 3 with diversifier 4 is
  really something.

12 Nov 2013.  Working on Kempe meet moves today.  Flag -O3 was
  removed in makefile so times above here are too long.  I
  changed Kempe meet moves so that now they will do anything
  that does not increase total irregularity.

  Had a bright idea and am now detaching timetable monitors when
  the last monitor that depends on them is detached.  Got a soln
  of the same cost in 9.02 secs, slightly better than 9.11.

13 Nov 2013.  Yesterday's change to Kempe meet moves had a bug
  whose effect was to not preserve regularity properly.  After
  fixing the bug the solution at the end of time assignment was

    KheGeneralSolve2014 after time assignment (6.63 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 2791.00066)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                 666    2787.00000
      SpreadEventsMonitor                    58       0.00066
      OrdinaryDemandMonitor                   4       4.00000
      -------------------------------------------------------
      Total                                 728    2791.00066
    ]

  and at the end was this, rather unluckily:

    KheGeneralSolve2014 at end (10.01 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 10.00524)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   4       9.00000
      AvoidSplitAssignmentsMonitor           36       0.00430
      SpreadEventsMonitor                    35       0.00036
      AvoidClashesMonitor                     1       1.00000
      LimitBusyTimesMonitor                  23       0.00058
      -------------------------------------------------------
      Total                                  99      10.00524
    ]

  The best of 8 was this after time assignment:

    KheGeneralSolve2014 after time assignment (3.14 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 4, cost 2789.00027)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                 666    2787.00000
      SpreadEventsMonitor                    27       0.00027
      OrdinaryDemandMonitor                   2       2.00000
      -------------------------------------------------------
      Total                                 695    2789.00027
    ]

  and this in the end (the whose set is 5.00464, 7.00497, 8.00658,
  8.00658, 10.00524, 10.00524, 11.00353, 15.00665):

    [ Soln (instance "AU-BG-98", diversifier 4, cost 5.00464)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   2       3.00000
      AvoidSplitAssignmentsMonitor           37       0.00410
      SpreadEventsMonitor                     2       0.00002
      AvoidClashesMonitor                     1       1.00000
      AvoidUnavailableTimesMonitor            1       1.00000
      LimitBusyTimesMonitor                  24       0.00052
      -------------------------------------------------------
      Total                                  67       5.00464
    ]

  Ignoring zones when trying to fix demand defects produced this:

    KheGeneralSolve2014 after time assignment (2.67 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 2792.00026)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                 666    2787.00000
      SpreadEventsMonitor                    26       0.00026
      OrdinaryDemandMonitor                   5       5.00000
      -------------------------------------------------------
      Total                                 697    2792.00026
    ]

    KheGeneralSolve2014 at end (6.46 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 5.00527)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   2       4.00000
      AvoidSplitAssignmentsMonitor           39       0.00450
      SpreadEventsMonitor                    11       0.00011
      AvoidClashesMonitor                     1       1.00000
      LimitBusyTimesMonitor                  29       0.00066
      -------------------------------------------------------
      Total                                  82       5.00527
    ]

  which is nothing remarkable.  Best of 8 (4.00542, 5.00513,
  5.00527, 5.00527, 8.00662, 8.00662, 8.00737, 9.00623) was

    KheGeneralSolve2014 after time assignment (3.87 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 5, cost 2789.00022)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                 666    2787.00000
      SpreadEventsMonitor                    22       0.00022
      OrdinaryDemandMonitor                   2       2.00000
      -------------------------------------------------------
      Total                                 690    2789.00022
    ]

    [ Soln (instance "AU-BG-98", diversifier 5, cost 4.00542)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   1       2.00000
      AvoidSplitAssignmentsMonitor           43       0.00470
      SpreadEventsMonitor                    12       0.00012
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  25       0.00060
      -------------------------------------------------------
      Total                                  83       4.00542
    ]

  Altogether I think we'll stick with the bug fix and try to
  do something about the unlucky first solution:

    KheGeneralSolve2014 after time assignment (6.78 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 2791.00066)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                 666    2787.00000
      SpreadEventsMonitor                    58       0.00066
      OrdinaryDemandMonitor                   4       4.00000
      -------------------------------------------------------
      Total                                 728    2791.00066
    ]

    KheGeneralSolve2014 at end (10.22 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 10.00524)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   4       9.00000
      AvoidSplitAssignmentsMonitor           36       0.00430
      SpreadEventsMonitor                    35       0.00036
      AvoidClashesMonitor                     1       1.00000
      LimitBusyTimesMonitor                  23       0.00058
      -------------------------------------------------------
      Total                                  99      10.00524
    ]

  which does after all have a pretty good time assignment; it
  goes wrong during resource assignment.  It very unluckily
  dropped two English double periods on top of the English
  faculty meeting and student welfare meeting.  There are no
  demand defects but still the two classes are unassignable.
  A simple move of the student welfare meeting to another
  time when everyone is available would free up two English
  teachers.  (Both have no available workload, but an ejection
  chain should be able to handle that.)

  I've started serious work on monitor grouping, doing event
  grouping at the moment.  I've just realized that grouping
  spread events and order events monitors is different from
  the others, since they monitor a set of events.  Two of
  these monitors need to know that they are monitoring one
  event from the same set of classes, if they are to be grouped.
  Also we still need classes_by_event.

14 Nov 2013.  Prefer times monitors have a duration field which
  may be used to prove that some of them have provably fixed zero
  cost, because the meet splits are fixed and the prefer times
  monitors do not apply to meets of their durations.

  Got grouping of assign time and prefer times monitors working.
  Could easily change things so only attached ones are grouped.
  Must think about that.

  Grouping of spread events monitors seems to be working.  I've
  also written grouping of order events monitors, but I can't
  test that.  Anyway it's running.

15 Nov 2013.  Now detaching prefer times monitors when their
  duration attributes allow it.

  Have a firm spec for KheSolnStandardGroupMonitors, must
  implement it.  But it seems that multiple parents should
  come first.  Have specified them now too, and just begun
  to implement them, using new type KHE_MONITOR_LINK.

16 Nov 2013.  Got a clean compile with the new multiple parents code.
  Found a bug - khe_soln.c was not inheriting KHE_MONITOR_INHERIT.  It
  is now inheriting KHE_GROUP_MONITOR_INHERIT, which gives it all the
  group monitor stuff except the copy field.  Also removed an old
  trace_cost field from monitors, no longer used.

17 Nov 2013.  Finished implementing KheSolnStandardGroupMonitors,
  including working out and documenting how the standard grouping
  should treat all kinds of monitors wrt linkage to the solution.
  Currently running to the end, with ejection chain time repair
  turned off.  Solution is naturally pretty bad.

18 Nov 2013.  Revised the ejection chains code to have two group
  monitors, one for the main loop and one for augment functions,
  and also to pass a trace object to each augment function.  I've
  also revised the ejections chains chapter accordingly, although
  not the time repair and resource repair sections yet; I will
  need to revise their functions first.  Also worked over the
  later parts of the grouping chapter.

19 Nov 2013.  Implemented KheSolnBypassAndDeleteAllGroupMonitors,
  finished the revision of the later part of the grouping chapter,
  and have a clean compile of most of it.

20 Nov 2013.  Auditing yesterday's work.  Decided to break up the
  standard grouping into its split, time, and resource parts; this
  gave a better basis for thinking about grouping demand monitors,
  which I did and came up with these uses for grouping them:

    Kempe meet moves:  a quite separate secondary grouping.
    Time assignment: primary grouping by first unfixed meet.
    Resource assignment: grouped into a single monitor to pass
      as a limit monitor to the resource ejector.

  Documented, implemented, and audited all this.

21 Nov 2013.  Wrote the last two secondary grouping functions,
  and made sure that the primary and secondary groupings were
  all being created and deleted at the right places.  This
  included documenting, implementing, and using primary
  grouping delete functions.

22 Nov 2013.  Changed the Augment interface to include a scratch
  transaction object.  Split up the old augment functions into
  the forms needed for the new groupings.  Worked on the ejection
  chains chapter.

23 Nov 2013.  Worked on the resource repair section of the ejection
  chains chapter, and audited the code at the same time.

  Changed the primary grouping of prefer times and prefer resources
  monitors to require that they request the same sets of times (or
  resources) as well.  Implemented and documented.  Also rewrote
  all group monitor repair functions to ensure that they uniformly
  try repairing the first monitor on their defect list.

  Sorted out a problem with avoid split assignments repairs not
  taking fixed assignments into account as they should.  Also,
  am now attempting to repair only the assigned tasks, since
  these are the ones producing the defect, and assigning the
  unassigned ones as well is overkill and can be left to the
  assign resource defect repairer.  Each repair is a set of
  Kempe task moves.

24 Nov 2013.  Worked on the resource solvers chapter and tidied
  up Kempe resource repair:  I now have two operations, called
  Kempe-step task assignment and Kempe-step task move.

  To access clashes, added KheTimetableMonitorClashingTimeCount
  and KheTimetableMonitorClashingTime.

  Done something new at last: KheResourceRepairAvoidClashesAugment
  and KheResourceRepairAvoidUnavailableTimesAugment, also redone
  KheResourceRepairLimitWorkloadAugment (overload case only).

25 Nov 2013.  Updated time group monitors and their clients to
  pass the time group across, updated limit busy times monitors
  to store a list of defective time groups, and targeted limit
  busy times repairs to the defective time groups.  Updated the
  documentation, which is in pretty good shape now.  Just avoid
  split constraints to sort through before I can start testing.

26 Nov 2013.  Sorted through task code and made assignment,
  unassignment, move, and swap clearer and more efficient.
  Should really do the same for meets, but I can't be bothered
  just now.  Also checked the Kempe step code and documentation,
  they seem to be fine now that they are built on a clearer
  foundation.  Also checked the resource ejection chain stuff,
  and fixed up KheResourceRepairAvoidSplitAssignmentsAugment.

  Started testing, fixed a few simple bugs, got a quite fast
  run but quality of time assignment (at least) is poor, so
  the next step is to look into that.

27 Nov 2013.  The problem yesterday was that the final time
  assignment ejection chain (the one without regularity) was
  turned off.  Turning it on produced a pretty good time
  assignment (4 demand defects), but a dreadful resource
  assignment.  Looking into that now.

  Everything seems to be working, just bad luck I suspect
  that things turned out so bad.  One thing I could try
  is to break resource assignment into two bits and delay
  untightening domains until after the first bit.  This
  might encourage better resource assignments to begin with.

29 Nov 2013.  Took yesterday off.  Discovered that I introduced
  a bug into KheTaskMoveCheck when tidying it up, and that this
  had prevented many task moves that should have worked.  Fixing
  that got me back into the usual ballpark:

    KheGeneralSolve2014 at end (17.92 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 11.00451)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   6       9.00000
      AvoidSplitAssignmentsMonitor           33       0.00390
      SpreadEventsMonitor                    11       0.00011
      AvoidClashesMonitor                     2       2.00000
      LimitBusyTimesMonitor                  22       0.00050
      -------------------------------------------------------
      Total                                  74      11.00451
    ]

  Best of 8:

    [ Soln (instance "AU-BG-98", diversifier 3, cost 3.00429)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   1       2.00000
      AvoidSplitAssignmentsMonitor           35       0.00370
      SpreadEventsMonitor                    11       0.00011
      AvoidClashesMonitor                     1       1.00000
      LimitBusyTimesMonitor                  22       0.00048
      -------------------------------------------------------
      Total                                  70       3.00429
    ]

  Hurrah!  best so far.  The 8 costs were not so great:  3.00429,
  7.00623, 11.00451, 11.00557, 6.00460, 8.00606, 11.00451, 11.00557.

  After adding a bit more diversification in time assignment, the
  8 values were 15.00569, 7.00623, 11.00451, 10.00540, 6.00460,
  8.00576, 11.00451, 15.00569, which is perhaps a more realistic
  statement of where we are.

  Added diversity to the ejection chain solver, by sorting the
  defects in the main loop slightly differently depending on
  the diversifier.  Final costs are (5.00551, 8.00516, 8.00516,
  10.00671, 11.00594, 11.00594, 12.00395, 13.00598) which still
  does not seem very diverse.  However, let's stick with it.

  Ran the grouping/ungrouping 10000 times, and times before and
  after were 0.18 and 31.53 seconds, so the time per call was

      (31.53 - 0.18) / 10000 = 0.003 seconds

  I've documented this, saying that it's not significant if
  done only occasionally.

  Thinking again about vizier nodes.  Without one, result was

    KheGeneralSolve2014 at end (21.31 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 8.00516)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   4       7.00000
      AvoidSplitAssignmentsMonitor           43       0.00460
      SpreadEventsMonitor                    10       0.00010
      AvoidClashesMonitor                     1       1.00000
      LimitBusyTimesMonitor                  20       0.00046
      -------------------------------------------------------
      Total                                  78       8.00516
    ]

  With a vizier node, result was

    KheGeneralSolve2014 at end (17.22 secs so far):
    [ Soln (instance "AU-BG-98", diversifier 0, cost 5.00640)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   3       5.00000
      AvoidSplitAssignmentsMonitor           46       0.00520
      SpreadEventsMonitor                    49       0.00074
      LimitBusyTimesMonitor                  21       0.00046
      -------------------------------------------------------
      Total                                 119       5.00640
    ]

  Faster and better!  8 with vizier node:  (4.00418, 5.00607,
  5.00640, 5.00640, 6.00507, 10.00538, 12.00561, 12.00561) as
  compared with (5.00551, 8.00516, 8.00516, 10.00671, 11.00594,
  11.00594, 12.00395, 13.00598) without the vizier.  Clearly
  the vizier is doing well here.  Perhaps we should install
  it only after the last layer, and use it on the last repair?

30 Nov 2013.  Written a little stats module and moved timer
  code into it and made it public.  Written code to generate
  a table whose rows are (instance, cost, time), and am
  currently trying to produce this table for XHSTT-2013.

  Fixed a bug in KheTailHasRepeats which crashed solving
  DK-HG-12.  The fix did not change my results on BGHS98.
  But I then found several problems with DK-HG-12, which were
  producing 411 unmatched demand tixels before doing anything
  at all.  I've documented the problems in to_gerhard01 and
  sent them off to him for comment.  Meanwhile, I'll get my
  code to skip instances with that name.

1 Dec 2013.  Roughed out the Experiments section of the paper
  so that I can see what experiments I have to do.  Now working
  on a much more elaborate tables package.

2 Dec 2013.  Finished off the revised stats package (except I
  haven't done the LaTeX format at all yet), ready to use it.
  Started work on adding options, have a framework for all
  and have finished the interfaces of time options, but still
  have to add the actual options.

3 Dec 2013.  Got the interfaces of the solvers right, and did four
  options (diversify, resource_invariant, resource_rematch, and
  two_colour_repair).  Changed the name TwoColourRepair to
  ResourcePairRepair, to make clear it's a resource solver.

4 Dec 2013.  Did some reorganizing of the user's guide, and added
  the complete set of options.  Ran solver again, got 6.00512.
  Found bug in KheResourcePairReassign.

5 Dec 2013.  Re-implemented KheResourcePairReassign today, all
  done except a bug fix is needed in KheInvariantTransactionBegin
  and KheInvariantTransactionEnd.  The old spec and implementation
  seem to have been rather dodgy:  they were adding things when
  they should have been merging them.  One run fixed 8 split
  assignments this way; its final cost was 6.00462, including 35
  split assignments defects, which is fewer than we were getting
  above.  Resource rematching was also running but it always
  failed to find an improvement.

6 Dec 2013.  Sorted out the issues with KheInvariantTransactionBegin
  and KheInvariantTransactionEnd; replaced them with new and better
  functions KheAtomicTransactionBegin and KheAtomicTransactionEnd.
  No more crashing now, best of 8 gave 6.00462.

  Back to working on code for gathering statistics, have begun a
  serious attempt to get the ejection chain statistics going.

7 Dec 2013.  Working on augment types and repair types.  Have
  documented a set of simple and effective statistics to collect,
  and added stubs for their calls to khe_sm_ejector.c.  Also
  added and documented global flag KHE_USE_TIMING.

9 Dec 2013.  Implemented all the new statistics functions.

10 Dec 2013.  Started working top-down today, with command line flag
  "khe -g<num> archive" for running test number <num> on archive,
  and a detailed specification of KheSolveForTesting.

  Sorted out parallel solving:  both functions use the same
  underlying implementation now.

11 Dec 2013.  While getting KheBenchmark() working for test 1,
  I saw this debug output (best of 8):

    [ Soln (instance "AU-BG-98", diversifier 4, cost 3.00517)
      Soln                              Defects          Cost
      -------------------------------------------------------
      AssignResourceMonitor                   2       2.00000
      AvoidSplitAssignmentsMonitor           43       0.00460
      SpreadEventsMonitor                     9       0.00009
      AvoidClashesMonitor                     1       1.00000
      LimitBusyTimesMonitor                  21       0.00048
      -------------------------------------------------------
      Total                                  76       3.00517
    ]

  This is pretty good.

  Added KheArchiveAddInstance and KheArchiveDeleteInstance, to
  make it easy to reorganize archives.  Then added a -E flag
  to explode an archive into one archive per country.  I can
  now use AU.xml as a relatively fast test.

  Brought the stats module into somewhat better shape; each
  file can contain multiple tables now.  And after fixing a
  dreadful bug I'm now generating two tables (Test 1) and
  incorporating them into the paper.  All good except a
  problem with the Average line.

12 Dec 2013.  Did a major rewrite of the table generator today,
  it's more flexible now and fixes the problem with average
  cost that I found yesterday.  Also highlights entries etc.
  All working.

  Did vizier node and full repair variant tests, working well.

13 Dec 2013.  Testing the cost of gathering statistics while
  running ejection chains.  With KHE_EJECTOR_STATS 1, the
  times were 

    Instance  -VZ-FR  -VZ+FR  +VZ-FR  +VZ+FR
    AU-BG-98    10.2     7.0     8.0    11.0
    AU-TE-99     1.7     2.4     1.7     2.9
    Average      6.0     4.7     4.9     6.9

  With KHE_EJECTOR_STATS 0, the times were

    Instance  -VZ-FR  -VZ+FR  +VZ-FR  +VZ+FR
    AU-BG-98    10.3     7.0     8.1    11.1
    AU-TE-99     1.7     2.4     1.7     2.9
    Average      6.0     4.7     4.9     7.0

  These are basically indistinguishable.

  Got a few more tests working, including finding lower bounds
  on room assignment that had a few twists and turns.

14 Dec 2013.  Although I got the lower bound on room assignment
  working yesterday, it was kludgy so today I'm redoing it to
  simply bail out after time assignment - much more transparent.

  Started on ejection tests.  KheEjectorSetSchedulesFromString
  is now more concise, so usable as a table column header.  Have
  clearly demarcated the constant parts of an ejector from the
  variable parts now.  I've also reorganized the two time and
  one resource ejection chain repair functions.  They are ready
  now to take their ejector objects from options, if available.

  Added graphs to the stats package.  Code for Test 6 is all
  written, needs testing now.

15 Dec 2013.  Did Test 6 (chain length histograms) and Test 7
  (augment function effectiveness) today, both look very good.

16 Dec 2013.  Got the repair operation effectiveness table going.

17 Dec 2013.  Added a test of encouraging node regularity, and
  did most of the work of tidying up resource pair repair.

18 Dec 2013.  Navigating the maze of resource pair repair.  I've
  cleaned up the interface, including the options; now I have to
  sort out the "interchangeable parts" mess.

  Generated tables showing the number of each kind of defect
  in each solution.

19 Dec 2013.  Attacking the "interchangeable parts" mess today.
  Making progress.

20 Dec 2013.  Wrote out a complete, concrete description of how
  resource pair repair can get rid of symmetries.  Now auditing
  the code to make sure the old parts agree with the new
  description, and implementing the new parts.

21 Dec 2013.  Making agonizingly slow progress with the resource
  pair repair rewrite.  I've just got it back to what it did
  before, only with separate objects for each part now.  And I
  have also managed to make those parts into nodes.  So I'm
  ready for the symmetry code.

22 Dec 2013.  Done quite a lot of useful tidying up in resource
  pair reassignment, including making sure that it handles
  unassignable components correctly, and improved the
  documentation accordingly.  Working on symmetry now; I've
  done rp-symmetry, so s-symmetry is next.

  Wrote module khe_two_colour.c in an attempt to come to grips
  with the problem.  It's working and in use by the resource
  pair code now.  That has simplified things somewhat, and
  should help with symmetry, which is next.

23 Dec 2013.  Tidied up the new module and documented it, and
  revised the documentation of how the main function is
  implemented.  I'm up to the crux now:  implementing
  symmetry removal using another call on the partition graph.
  Have a sketch of the algorithm on paper, need to sleep on
  it.  I also reorganized the PATAT paper so that the tables
  are spread evenly through it, rather than being clustered
  at the end.  Big improvement.

24 Dec 2013.  Wrote out a clear and concrete description of the
  symmetry removal algorithm; it's implemented but not tested.
  Off-site backup today.

26 Dec 2013.  Tested and debugged resource pair repair today.
  It seems to be working now, but it has been touch and go.

27 Dec 2013.  Rearranged the tables to include cost and time
  in the same tables wherever possible.  Did a complete run
  of all tests and incorporated the new results.  All good.

28 Dec 2013.  Extracted some interesting resource pair repair
  statistics, showing that repairing split assignments always
  searches the whole tree, but repairing generally sometimes
  fails to.  I think that's enough on that subject.  I started
  it on 17 December, which is about a week and a half's work.

  Did a complete run of all instances.  The results are not
  wonderful.  I think the next things to look at are IT-I4-96
  and UK-SP-06, which have large numbers of assign time defects.
  These seem odd, given that KHE14 is supposed to assign a time
  to every meet.  And IT-I4-96 has a solution of cost 0.00057
  according to Gerhard's web site.

  Started looking at IT-I4-96.  There is a very long pause
  before it even starts assigning time layers, have to look
  into that.

  One problem is the DD spread events constraint, it is being
  taken to mean that the DD meets are part of one course, which
  they evidently aren't.  Should ignore it if it makes nodes
  too big.  Also these are what I used to call "single-resource
  meetings", they should not be assigned times early on.

  Found a bug in KheEventSharePreassignedResource which was
  causing the layer tree to merge the DD meets into one node
  when it shouldn't have.

29 Dec 2013.  Had a bright idea for beefing up the student
  sectioning paper that has been on the backburner for a
  couple of years now.  Spent the day working on that.  It
  now looks good enough to submit.

30 Dec 2013.  Back at work on IT-I4-96.  The basic problem
  with the solve at the moment is that resource defects are
  being introduced during time assignment, owing to the many
  preassigned resources (teachers, they are), and nothing is
  being done to repair them.  So I've begun by adding all
  resource monitors to the list of children of the group
  monitors used by time repair.  I now need to write augment
  functions for each of these, or at least the worst offenders.
  I've done one for avoid unavailable times defects, not tested
  yet but it was quite easy really, basically a cut-and-paste
  job.  I should work over the code to unify things a bit.

31 Dec 2013.  Made good progress unifying the time repair
  ejection chains code (khe_st_ejection.c).  The idle times
  repair is half written, finishing it is the next step.

