Merge nucleic/sleek-ember-seal-uady into dev
This commit is contained in:
@@ -554,6 +554,34 @@ distillation regression cannot overwrite the 80.31% candidate. Teacher agreement
|
||||
reported for diagnosis but does not enter deep checkpoint selection; overall and hard
|
||||
primary label accuracy remain the only selection inputs.
|
||||
|
||||
### Pause and resume purpose-deep training
|
||||
|
||||
`train_deep_mlx.py` handles both `Ctrl-C` (`SIGINT`) and `SIGTERM` gracefully. Press
|
||||
`Ctrl-C` once: the current optimizer step finishes, then the trainer atomically saves the
|
||||
current model, AdamW state and learning-rate cursor, epoch and next batch, exact shuffle
|
||||
order, accumulated losses, RNG state, best-selection state, and history under
|
||||
`<output-dir>/resume/`. It exits with the conventional signal-derived status only after
|
||||
that checkpoint is durable. A second `Ctrl-C` forces an immediate interruption.
|
||||
|
||||
The pause message prints the complete resume command. For the distilled run above it is:
|
||||
|
||||
```bash
|
||||
ml/purpose-classifier/venv/bin/python -u \
|
||||
ml/purpose-classifier/train_deep_mlx.py \
|
||||
--resume-training \
|
||||
ml/purpose-classifier/outputs/purpose-deep-v1-base-mlx-distilled/resume
|
||||
```
|
||||
|
||||
`--resume-training` is sufficient by itself: it restores the original dataset,
|
||||
distillation, schedule, loss, and selection arguments from the checkpoint. It also verifies
|
||||
hashes of the training split, validation split, and teacher cache before loading. The
|
||||
checkpoint resumes at the next unprocessed batch—even if stopped after the last training
|
||||
batch but before epoch evaluation. Normal completion removes the large optimizer resume
|
||||
artifact while retaining the selected `model/` and reports.
|
||||
|
||||
This applies to processes started with the updated script. A process already running older
|
||||
code cannot acquire signal handling retroactively.
|
||||
|
||||
The historical large-rung rule required base to qualify first, large to beat base by at
|
||||
least two hard-slice points, and deep to reach 97% scored overall while beating lite by
|
||||
five hard-slice points. Neither the v1 nor Sol-high v2 base result unlocked that rung.
|
||||
|
||||
Reference in New Issue
Block a user