LP-SFT: Keeping Plausible Alternatives Alive in Supervised Fine-Tuning

Paper: LP-SFT: Keeping Plausible Alternatives Alive in Supervised Fine-Tuning (v4) https://arxiv.org/html/2607.04733v4

LP-SFT uses a frozen base model as a local structural reference during supervised fine-tuning. Its preservation objective concerns selected non-label candidates, while ordinary cross-entropy still learns the observed response.

The motivation comes from structured next-token distributions. Plausible continuations can form near-uniform candidate regimes at different effective support sizes. Entropy analysis describes these patterns across instruction, code and mathematics data. Treating all non-label possibilities as uninformative tail noise can discard that structure.

The method first selects highly ranked base candidates and excludes the supervised target. It then normalizes probabilities within the remaining local set before applying a KL comparison. Removing the target avoids a direct conflict with label supervision; normalization lets the label grow without requiring the alternatives to keep their total probability mass.

A toy distribution makes the latter point concrete. Alternatives at 30%, 20% and 10% become 15%, 10% and 5%. The total falls from 60% to 30%, while relative proportions remain 3:2:1. Those authored values explain the goal, not a measured model update or a hard constraint imposed by the training loss.

Base candidate logits can be computed offline because the reference model and training examples are fixed. The subsequent fine-tuning loop uses cached information, avoiding a second frozen-model forward pass at every step. One-time cache preparation remains part of the computational account.

Table 1 evaluates a mixed-domain Qwen3-14B configuration. The average of displayed metrics is 66.79 for the base, 58.55 for cross-entropy and 65.32 for LP-SFT. MMLU scores are respectively 80.51, 77.92 and 80.05. These comparisons show retention closer to the base under LP-SFT; they do not turn the aggregate into pooled task accuracy or establish that this configuration exceeds its base model.

Pinned source versions: Paper, v4: https://arxiv.org/html/2607.04733v4

#FineTuning #LanguageModels #AIResearch