Merge nucleic/sleek-ember-seal-uady into dev

This commit is contained in:
2026-08-02 22:44:24 -07:00
parent ae616c6f85
commit 1a41febf73
4 changed files with 673 additions and 146 deletions
+28
View File
@@ -554,6 +554,34 @@ distillation regression cannot overwrite the 80.31% candidate. Teacher agreement
reported for diagnosis but does not enter deep checkpoint selection; overall and hard
primary label accuracy remain the only selection inputs.
### Pause and resume purpose-deep training
`train_deep_mlx.py` handles both `Ctrl-C` (`SIGINT`) and `SIGTERM` gracefully. Press
`Ctrl-C` once: the current optimizer step finishes, then the trainer atomically saves the
current model, AdamW state and learning-rate cursor, epoch and next batch, exact shuffle
order, accumulated losses, RNG state, best-selection state, and history under
`<output-dir>/resume/`. It exits with the conventional signal-derived status only after
that checkpoint is durable. A second `Ctrl-C` forces an immediate interruption.
The pause message prints the complete resume command. For the distilled run above it is:
```bash
ml/purpose-classifier/venv/bin/python -u \
ml/purpose-classifier/train_deep_mlx.py \
--resume-training \
ml/purpose-classifier/outputs/purpose-deep-v1-base-mlx-distilled/resume
```
`--resume-training` is sufficient by itself: it restores the original dataset,
distillation, schedule, loss, and selection arguments from the checkpoint. It also verifies
hashes of the training split, validation split, and teacher cache before loading. The
checkpoint resumes at the next unprocessed batch—even if stopped after the last training
batch but before epoch evaluation. Normal completion removes the large optimizer resume
artifact while retaining the selected `model/` and reports.
This applies to processes started with the updated script. A process already running older
code cannot acquire signal handling retroactively.
The historical large-rung rule required base to qualify first, large to beat base by at
least two hard-slice points, and deep to reach 97% scored overall while beating lite by
five hard-slice points. Neither the v1 nor Sol-high v2 base result unlocked that rung.