Building Cybersecurity Language Models with MiST

At Dream, we build specialized models for cybersecurity and sovereign AI deployments, where sensitive data may need to remain within controlled infrastructure. MiST explores how to give compact language models stronger cybersecurity capabilities through a dedicated mid-training stage.

Our paper, MiST: Mid-Training LLMs for Cybersecurity, introduces 8B and 32B models adapted from Qwen checkpoints. We turn a curated cybersecurity seed corpus into synthetic explanations, Q&A and analysis, then use that data before supervised fine-tuning and preference tuning.

We are releasing MiST-8B on Hugging Face for cybersecurity research, further specialization and deployment within your own infrastructure.

Across the paper’s 13-task cybersecurity suite, MiST-8B-DPO raises the mean score from 48.6 to 61.7, a 13.1-point gain over Qwen3-8B. MiST-32B-DPO scores 63.2, versus 54.6 for Qwen3-32B. The 8B model also outperforms the open cybersecurity baselines included in our evaluation, including larger models.

What mid training adds

Mid-training adapts a pretrained model to the target domain before post-training. Like raw continual pre-training (CPT), it uses next-token prediction. MiST focuses on how the training corpus is constructed: expert resources are transformed into examples that explain cybersecurity concepts and their relationships.

This matters because much of cybersecurity knowledge is stored in compact, structured records. An NVD vulnerability record in our seed corpus averages about 150 tokens, often combining a short description with a severity vector and a weakness mapping. Synthetic transformations expand these records into material that explains what those fields mean and how they relate.

Supervised fine-tuning (SFT) then teaches the chat format and instruction following while strengthening cybersecurity capabilities. Direct preference optimization (DPO) uses general preference data to refine response behavior. Nearly all of the cybersecurity gains are already present after mid-training and SFT.

MiST-8B starts from Qwen3-8B-Base. MiST-32B starts from a post-trained Qwen3-32B checkpoint because a base checkpoint was unavailable, so the 8B experiments provide the more controlled test of the recipe.

We are also releasing MiST-30B-A3B, based on NVIDIA’s Nemotron 3.5 Lightning 30B-A3B base model. It uses the same mid-training and SFT stages and training data as MiST-8B, without the final DPO stage.

Building the training corpus

We begin with about 220,000 curated documents totaling 65 million tokens. Sources include NVD vulnerability records, MITRE taxonomies, NIST and OWASP guidance, detection rules and threat-intelligence reports.

Qwen3-30B-A3B generates the synthetic data with source documents supplied in context. Four complementary flows build the mid-training corpus:

Paraphrases restate technical content in varied wording while preserving its meaning.

Educational rewrites turn a source document into a textbook-style explanation or practitioner article.

Question chains generate short factual Q&A pairs, followed by two rounds of questions covering aspects not addressed earlier.

Security analysis explains supplied metadata, such as CVSS vectors or ATT&CK technique IDs, instead of asking the model to invent those fields.

A verifier scores each generated sample on six criteria, including factuality, instruction adherence and coherence. Samples scoring below 8 out of 10 overall are removed. The generator and verifier use the same model, so this filter reduces errors without guaranteeing correctness.

The resulting mid-training corpus contains about one billion tokens, including 50,000 general instruction samples intended to help preserve broad capabilities. Separate conversations and persona-based chats provide cybersecurity data for SFT, alongside general instruction data.

Results across the cybersecurity suite

We evaluate 13 tasks from six public benchmarks: CTI-Bench, SECURE, AthenaBench, PrimeVul, MMLU-Cyber and SecEval. Paper results are averaged over five runs; MiST-30B-A3B and its Nemotron baseline use three runs. Reasoning or thinking modes are disabled where available. The table reports the unweighted mean across all 13 tasks.

Sources: MiST paper, Table 1; MiST-30B-A3B model card (MiST-30B-A3B and its Nemotron baseline). Selected models; scores are on a 0–100 scale.

MiST-30B-A3B scores 60.3, compared with 51.5 for its Nemotron base, an 8.8-point gain. These additional results are reported in its model card, separately from the paper’s Qwen-based experiments.

Full evaluation results are available in the MiST-8B and MiST-30B-A3B model cards.

One of the largest gains is in extracting attack techniques from threat-intelligence text: Qwen3-8B scores 9.4, compared with 57.3 for MiST-8B-DPO.

Where the gains come from

Controlled comparisons separate the contribution of mid-training from SFT and test the composition of the corpus. All scores below use the same 13-task suite:

Beyond SFT alone. With identical SFT, Qwen3-8B-Base scores 57.6 without mid-training and 62.4 with it, a 4.8-point gain.

Beyond raw CPT. Training on about 2.4 billion raw cybersecurity tokens, followed by the same SFT, scores 53.2. MiST-8B-SFT scores 62.4 using a mid-training corpus of about one billion tokens.

Value of the full mix. The best single generation flow scores 60.3, versus 62.4 for all four flows. Removing any one flow also lowers the mean.

These comparisons support the value of constructing a varied synthetic corpus for cybersecurity adaptation. Estimated training time for the full 8B pipeline is 7.1 hours on eight NVIDIA B200 GPUs, excluding data generation and filtering.

General capabilities were largely preserved for MiST-8B. MiST-8B-DPO averages 85.8 across MMLU, ARC-Challenge, GSM8K and IFEval, versus 82.5 for Qwen3-8B. The improvement is not uniform: instruction-following performance on IFEval falls slightly.

On the same four general-capability benchmarks, MiST-30B-A3B averages 81.0, versus 67.8 for its Nemotron base. Gains on GSM8K and IFEval outweigh a decline on ARC-Challenge, while MMLU remains roughly flat.

MiST also provides a useful starting point for further specialization. In the paper’s reinforcement-learning experiments on three structured cybersecurity tasks, MiST-8B-DPO achieves higher validation accuracy throughout training than Qwen3-8B.

These evaluations cover cybersecurity knowledge and structured analysis; they do not establish performance on autonomous, multi-step security workflows. The evaluated checkpoints are research models, separate from Dream’s production models. Data decontamination reduces benchmark overlap, but cannot rule out paraphrased leakage.

Next steps

Download MiST-8B and MiST-30B-A3B on Hugging Face to get started. Their model cards provide full evaluation results and usage instructions. Our paper provides the training recipe, ablations and evaluation details.

We have since applied MiST to NVIDIA Nemotron 3.5 Super VL, as described in our follow-up post. Future work will explore multi-step security workflows and other specialized domains.

We will present the work at EMNLP 2026 in Budapest, October 24–29.