Summary. We study an alternative training procedure that replaces one long, tightly synchronized training run with many short, independent training runs on distilled data, followed by prediction aggregation. Empirically, this yields up to approximately 4× reduction in wall-clock training time and up to approximately 3× reduction in compute cost relative to standard data-parallel baselines, while maintaining comparable performance on both image classification and language modeling workloads.
Training large neural networks is constrained by two factors: the number of sequential optimization steps required to reach good performance, and the overhead of coordinating work across many devices. Standard distributed training reduces wall-clock time by parallelizing gradient computation, but introduces communication bottlenecks and often relies on large effective batch sizes that can degrade optimization.
Our approach targets the sequential component directly. Instead of parallelizing individual gradient steps, we reduce the number of sequential updates required and shift computation into independent workers. Each worker trains a model on a short, distilled sequence of data, and the resulting models are aggregated at the end.
A Useful Mental Model
A large fraction of training updates are incremental refinements: once a model has captured the dominant structure in the data, additional updates tend to reinforce existing patterns rather than introduce fundamentally new ones. This suggests that long training runs contain redundancy.
The core idea is therefore to replace one long training trajectory with many shorter ones that capture the most informative parts of the dataset. While any individual short run is incomplete, averaging across many such runs recovers a stable approximation to the result of full training.
Method Overview
for i = 1, ..., N (in parallel):
initialize model θ_i
sample a distilled sequence S_i from dataset D
for k = 1, ..., L:
update θ_i on S_i[k]
compute predictions y_i
return average prediction (1/N) * Σ_i y_i
The procedure has two key components. First, each worker trains on a short, distilled sequence that preserves the most informative aspects of the dataset, reducing sequential depth. Second, predictions are averaged across workers, which stabilizes the result and compensates for the incompleteness of individual runs.
Workers do not communicate during training. Unlike standard data parallelism, there is no repeated synchronization of gradients or parameters—coordination happens only once, at aggregation.
Key Findings
We compare this procedure to standard data-parallel training on two workloads: image classification with ResNet-50 on ImageNet, and language modeling with GPT-2 on the Pile. In both cases, we measure the compute required to reach a fixed target performance across different wall-clock time budgets.
On ImageNet, the method achieves up to approximately 2.5× lower compute cost at fixed training time, and up to approximately 4× faster training at fixed compute. These gains are most pronounced in regimes where communication overhead limits the scalability of standard data-parallel training.
We observe a similar pattern in language modeling, with up to approximately 3× reduction in compute. Notably, increasing the training-time budget does not substantially improve the efficiency of standard training, indicating that a significant portion of computation remains sequential under conventional approaches.
Limitations and Scope
The effectiveness of this approach depends on constructing distilled sequences that retain the relevant structure of the full dataset. If the distilled data are not sufficiently informative, short training runs will not approximate full training well.
More broadly, this should be viewed as identifying a useful scaling regime rather than eliminating sequential optimization entirely. Some sequential computation remains necessary, but the required depth can be reduced substantially in practice.
Implications
These results highlight an alternative axis for improving training efficiency: reducing sequential depth rather than only increasing parallel throughput. The resulting procedure is communication-light and well-suited to distributed or bandwidth-constrained environments.
More generally, the results suggest that a meaningful portion of training computation is compressible. Understanding when this compression is possible may be an important direction for future work in efficient training.