The Benefits of Training Small Models on Many Tasks

Improving SLMs with multi-task fine-tuning

The author is a data scientist at Dream

Research overview

• Fine-tuning on up to 24 tasks at once did not hurt per-task accuracy in most settings we tested
• Transfer to unseen related tasks improved with every task added, by as much as 10.8 points
• The gain was roughly 6x larger in the smallest model (0.6B) than in the largest (8B)
• Related capabilities belong in one training run, not one model each

We fine-tuned small and mid-sized language models (0.6B to 14B parameters) on as many as 24 classification tasks at once. In most of the settings we tested, adding tasks did not reduce accuracy on those tasks. Performance on related tasks the model had never seen actually improved as more tasks were added.

That increase reached 10.8 percentage points at N=24, and it was more pronounced as the models got smaller. The gain was roughly six times larger in the smallest model we tested than in the largest.

Small models that power national capabilities

Many of our national sovereign AI deployments run entirely on-premise, inside agentic flows that make many model calls per operation. Using a frontier model for every step would be slow, expensive, and in some environments untenable. Our team frequently takes off-the-shelf models small enough to train on a single GPU and fine-tunes them for specific jobs and ecosystems: classifying attack techniques from a description, answering networking questions, or sorting cyber-defense event reports. like this purpose-built on-premise cyber-defense model.

In the real world, though, missions rarely represent only one task. A mission may need 5, 10, or 15 related classification jobs, plus a workflow to load, route between, and maintain a model for each.

My initial assumption was that given each model's limited capacity, training on more tasks should eventually make it worse at each one. I assumed we would be fine with a couple of tasks, and I wanted to find where it breaks.

In most of the settings we tested, it didn't.

What we actually tested

We ran seven experiment series, hundreds of sweeps overall, on broad-sense text classification built from public datasets.

The core measurement trained on N tasks, then evaluated two things: in-distribution accuracy, on held-out test sets of the tasks included in training, and out-of-distribution (OOD) accuracy, on tasks from the same pool that the model never saw in training.

We kept training samples per task constant as N grew, chose hyperparameters near a good single-task working point, and controlled for validation-set contamination. The design favors a fair comparison with single-task training, but it did not isolate task diversity from the effect of adding more total data. That limitation comes up again below.

More tasks rarely reduced performance on the tasks being learned

The most consistent result was that adding tasks usually did not dilute performance on the tasks included in training.

In the first cyber experiment (Qwen2.5-14B), best-configuration full fine-tuning scored 83.4% at N=1 and 83.2% at N=4-5, with LoRA nearly identical. At nine cyber datasets, accuracy moved from 70.3% with one task to 72.0% with nine. Seven legal datasets held flat around 70% from one task to six.

LegalBench was the useful exception. In the 25-task pool, in-distribution accuracy fell from 91.8% at one task to 89.7% at four, then recovered to 93.3% at 24. Part of the dip traced to a label-space issue we later fixed, but the rest persisted, and we do not yet have a complete explanation for it.

In-distribution accuracy vs. number of training tasks (N), one line per experiment series. The fine-tuning gain over the untrained base model is preserved from N=1 through N=24.

The result is more specific than "more tasks make trained tasks better." The model usually retained the full benefit of fine-tuning while learning a much broader set of jobs.

The larger gain appeared on tasks the model had not seen

The more surprising and promising result was transfer. Models trained on more tasks performed better on related tasks that were excluded from training, and the gain compounded with N.

OOD gain over the untrained base model by number of training tasks. Cyber transfer is large from N=1, while LegalBench takes off past N of about 8.

In cyber, OOD accuracy rose from 64.4% with one task to 67.5% with four to five under full fine-tuning, on top of an already large N=1 gain. Legal moved the same direction at a fraction of the size, gaining 2.7 points by three tasks, mostly in one related pair (ECtHR-A and ECtHR-B).

The widest sweep, the 25-task LegalBench pool on Gemma-4, improved OOD over the base model by 0.5 points with one task, 2.2 with four, 7.2 with 12, and 10.8 with 24. The curve still had not flattened where we stopped.

LegalBench clean-25 pool on Gemma-4. In-distribution (green) dips at N=4 before recovering, while out-of-distribution (red) climbs through N=24.

As the graphs show, these curves do not reflect a single universal scaling law. The pattern that holds is that transfer strengthens with the number of related tasks, while its size varies substantially by dataset.

Multi-task training was also less sensitive to learning rate

Accuracy was not the only thing that changed. Across nine learning-rate settings on eight cyber datasets, single-task OOD accuracy ranged from 45% to 58% depending on the rate, while four to five training tasks stayed within a narrower 58-61% band and beat single-task in all nine settings.

Our working hypothesis is that a multi-task batch contains fewer examples from any one task, acting like a lower effective per-task learning rate. Practically, it means less hyperparameter babysitting.

Smaller models had more to gain

My initial concern was the opposite, that only mid-sized models had the spare capacity for this. So we repeated the identical experiment manifest (126 training combinations, 504 runs) on four Qwen3 sizes, from 8B down to 0.6B, with size as the only variable.

The same eight-task, cyber-only training group improved cyber OOD by 4.1 points at 8B and 24.1 points at 0.6B, roughly six times the effect from the same data.

OOD accuracy by training group across four Qwen3 sizes. The smaller the model, the steeper the multi-task gain.

One plausible explanation is that smaller off-the-shelf models begin with less relevant knowledge, which leaves more room for a family of related tasks to build a useful domain representation. That is an interpretation, not an established mechanism, and smaller is not always better. At 4B, a single legal source reduced cyber OOD by 5.0 points before broader legal training erased the damage, and at 0.6B the first mixed group cost 1.0-1.5 in-distribution points before recovering.

Still, for deployments forced toward the small end by latency, memory, or environment constraints, the trade reads as favorable. The gain is largest exactly where the constraints kick in.

Where the result stops

These experiments support a middle ground between broad pretraining and one model per task. They do not show that one model should absorb every task.

• Scope was limited to broad-sense classification. We did not test open-ended generation, long-horizon reasoning, tool use, or full agent workflows.
• Domain and task composition mattered by an order of magnitude. Cyber's OOD lift reached 12-16 points by N=4-5 while legal topped out at 2.7. CEHA, where 5-6 tasks ask different questions about the same event reports, was a genuine counterexample: some of its tasks improved with N, some degraded, and one fell by as much as 24 points. More tasks does not always improve transfer.
• Cross-domain training did not clearly beat same-domain training. In the 8B cyber-and-law sweep, cyber-only training improved cyber OOD by 2.4 to 3.7 points versus 0.8 to 1.4 from law-only. If you know where your unseen tasks will land, train in that domain.
• Adding tasks also added total training data, because we held samples per task constant. The experiments measure the operational choice of adding complete task datasets, not a clean split of data volume from task diversity.
• Coverage was wide but not exhaustive. The size sweep covered one domain pair and one model family, and aggregates can hide a task that fails even when the mean improves.

Lessons for our future SLM tuning

When a mission needs several related capabilities, we will no longer default to a model per capability. Combining related tasks into one run kept per-task quality, improved behavior on tasks we did not anticipate, widened the hyperparameter sweet spot, and mattered most on the small on-premise models where our sovereign deployments need it most.

‍