Summary. We study whether data distillation can make protein structure prediction models substantially cheaper to train. For Boltz-1 at a target LDDT of 0.4, our scaling-law analysis estimates a median cost reduction of approximately 2100× at a fixed training time. The exact value of this cost reduction comes with large uncertainty: the interquartile range varies from 40× to 1.3×107. Nevertheless, we conclude that carefully distilled data can move the cost-time frontier by orders of magnitude.
Protein structure prediction is one of the clearest examples of what foundation models can contribute outside language. Given an amino-acid sequence, a structure prediction model estimates the three-dimensional shape into which the protein folds. These predictions can accelerate biological research and drug discovery, but the models themselves are expensive to train.
This makes protein models a demanding test of a broader question: how much of today's training cost is truly necessary, and how much can be eliminated by choosing the training data more carefully?
A controlled benchmark with Boltz-1
We study Boltz-1, a popular open-source protein structure prediction model. Our data is an equal mixture of structures from the RCSB Protein Data Bank and OpenFold. We evaluate prediction quality using LDDT, or Local Distance Difference Test, a standard measure of how closely a predicted local protein geometry matches the reference structure. Higher LDDT is better.
For the baseline, we train Boltz-1 using the default hyperparameters in the Boltz repository. We vary both the number of training epochs and the number of devices, producing runs with different wall-clock times and costs. Rather than compare against a single baseline run, we take the Pareto frontier: the least expensive way the standard training procedure can reach a given training time.
We fix a target LDDT of 0.4 and ask a concrete question: at the same wall-clock time, how much does each method cost to reach that target? Fixing the target and time prevents an apparently inexpensive method from benefiting simply because it trains longer or stops at lower predictive quality.
Changing the data, not the model
Our intervention is entirely on the data side. We apply data distillation to construct a much smaller, more informative training set, then train Boltz-1 on that distilled data. The model architecture and the prediction task remain unchanged.
Ordinary training repeatedly processes a large dataset, even though examples vary substantially in how much useful training signal they provide. Data distillation attempts to preserve the signal that most improves the model while discarding redundancy. If the distilled set succeeds, each unit of compute produces more progress: the model needs fewer training examples and fewer GPU-hours to reach the same LDDT.
Accounting for uncertainty
A finite set of training runs does not determine a cost-time frontier exactly. To avoid treating one fitted curve as ground truth, we place a Bayesian posterior over plausible scaling laws relating training time, training cost, and LDDT. For each scaling law in the posterior, we compute the cost required to reach 0.4 LDDT. The margins in the figure show the interquartile range of those estimates.
A large shift in the cost-time frontier
At a fixed training time, the posterior median implies that distilled training reaches 0.4 LDDT at approximately 2100× lower cost than the standard Boltz-1 procedure. Across the middle 50% of plausible scaling laws, the estimated reduction ranges from about 40× to 1.3×107.
The size of this interval is as important as its midpoint. These experiments support a large efficiency improvement, but they do not yet identify its magnitude precisely. In particular, 2,100× is a posterior median estimate at one accuracy threshold and a fixed training time, but not a guaranteed cost reduction.
Implications
The conventional route to better foundation models is to increase the amount of compute applied to the full training distribution. These results point toward a complementary route: make the training distribution itself far more efficient.
An important next step is testing whether the same gains persist across model and dataset scales. It is also important to understand what properties of protein data make such aggressive distillation possible, and whether those properties carry over to other foundation-model domains.