Mid-Training NVIDIA Nemotron 3.5 Super VL for Cybersecurity

How Dream builds deep security knowledge into a 120B open model on NVIDIA DGX B200, and what an AI agent and a researcher taught us about training it fast.

Dream builds national sovereign AI and cyber defense solutions for governments. As part of these efforts, our customers need specialized national and domain-specific models that can run fully sovereign on hardware they own without compromising performance.

To deliver both cyber defense, we built a full cybersecurity post-training stack, which includes mid-training, SFT, and RL, and that covers multiple use-cases. One such example is Hercules, the model behind Hero, our autonomous on-prem cyber defense research team, which ranked #1 on CyberGym at its release with 96.6% on Level 1.

This post offers a deep dive on the foundation of that stack: mid-training. This is the stage where the general model acquires the domain knowledge that every later stage builds on. We applied MiST, our internally developed cybersecurity mid-training method (arXiv:2609.18496, EMNLP 2026 Main Track), to the recently open-sourced NVIDIA Nemotron 3.5 Super VL, and trained it with NVIDIA NeMo AutoModel open library on two NVIDIA DGX B200 systems.

The results speak for themselves:

• +8.8 points on cybersecurity. Nemotron-3.5-Super-MiST averages 74.1 across 15 security knowledge benchmarks, against 65.4 for the original checkpoint. This improves on all 15 and leads every 120B-class open model we tested on 12 of them.
• General ability is intact. The average across GSM8K, MMLU, ARC-Challenge, and IFEval stayed the same.
• A small budget. Mid-training took 28.5 hours on 16 NVIDIA B200 GPUs.
• An AI agent increased training speed by 3.6×, after one redirect from an experienced researcher.

Figure 1 shows the result benchmark by benchmark. On every row our model shows an improvement relative to the original base model, while also achieving higher scores compared to other models of the same size.

Why mid-training

Cybersecurity is a complex domain, which requires highly specific knowledge. Unlike many other domains, cyber knowledge doesn't live in long prose. Much of the information is captured in compact, cross-referenced artifacts such as CVE records, CWE and ATT&CK taxonomies, detection rules, and threat reports that make for poor training text.

MiST treats mid-training as a corpus-design problem.

Figure 2 shows where it sits. NVIDIA's released Nemotron 3.5 Super VL already has general pre-training behind it. MiST adds two phases on top, with the resulting checkpoint functioning as the starting point for the rest of our stack:

• The first phase, continued pre-training, teaches the knowledge on a high-quality security corpus.
• The second, supervised fine-tuning, uses high-quality security conversations mixed with replayed general instruction data. It turns that knowledge into useful answers without eroding general ability.

Both phases ran with NeMo AutoModel on two DGX B200 systems.

Figure 3 shows how the security corpus is built. It starts from a seed corpus that our security experts vetted by hand and that includes vulnerability records, taxonomies such as CWE and MITRE ATT&CK, public reports, and defensive guidance. An LLM then expands each seed document through six generation flows, each designed to teach something different:

• Education rewrites a document as a textbook section or a practitioner's blog post.
• Conversations and persona-based chats (a SOC analyst, a red-teamer, a developer) show the knowledge used in context.
• QA + follow-up asks short factual questions, then targets what earlier rounds missed.
• Security analysis explains structured metadata, such as why a CVSS vector scores as it does or which tactic an ATT&CK technique serves.
• Paraphrases restate the same text in many forms, so the model learns concepts rather than verbatim text.

Every generated sample then passes a quality filter for coherence, factuality, relevance and instruction adherence. The surviving samples are combined with the original seed documents and a share of general data, and decontaminated against our benchmarks.

Why We Used Nemotron 3.5 Super

NVIDIA Nemotron 3.5 Super VL is a 120B-parameter hybrid Mamba-Transformer mixture-of-experts with 12B active parameters per token. That gives a powerful model at a cost a single DGX can serve efficiently. Critically for many of our government customers, it is both Western and open - NVIDIA publishes the weights along with the data and recipes behind the Nemotron family, so customers can inspect what they deploy and run it in an air-gapped data center. It also helpfully already has some cyber knowledge. The original checkpoint scores 65.4 on our security suite, level with Qwen3.5-122B-A10B. That makes it a hard test for MiST, with the challenge of improving a good model without breaking it.

Training fast: an agent, a researcher and 16 experiments

Our first two-node NeMo AutoModel recipe ran at 17.2K tokens/s, which put the full run at four and a half days. We handed the tuning to an AI coding agent in an auto research-like setting. Its brief was to find a configuration faster than 50K tokens/s, using short probe runs, with an experienced researcher monitoring the process.

Figure 4 tells the story. Each point is one experiment, left to right in the order it ran. The vertical line marks the moment the researcher stepped in. Red crosses at zero are running that crashed or ran out of memory. Grey circles ran but were not kept, and green circles are the changes that made it into the final recipe.

Phase 1: topology

For five hours the agent reshaped the parallel topology, meaning how the model is split across the 16 GPUs. That is the left half of Figure 4. Six runs crashed or ran out of memory, and the one new layout that ran ("fall back to CP 2") matched the baseline to within 1%. The agent kept going because nvidia-smi showed 95% GPU utilization, which looked like busy GPUs.

Why utilization misleads. Utilization counts the time any kernel is running, not how much work it does. A communication kernel waiting on the network counts as busy, and so does one spinning until its peers catch up. Power draw tells the truth: waiting is cheap, and matrix math is not. A profile showed 64% of GPU time going to communication and spin-waits. Measured by power, the GPUs drew about 430 W out of 1,000 W. They were mostly waiting.

The redirect. The researcher identified the pattern. Since two very different topologies gave identical throughput, topology wasn't the bottleneck. They instructed the agent to go after the kernels, communication and memory instead, and to measure power rather than utilization.

Phase 2: kernels, communication and memory

Within an hour the agent found the unlock, the first green point in Figure 4: a fused cross-entropy kernel that never materializes the full logits tensor. That freed the memory the better topology needed, and, as a result, throughput jumped to about 30K. Keeping experts resident in memory, reducing gradients in bf16 and using larger micro-batches took it to 62K, the steady climb on the right of Figure 4. Power rose in step, from 430 W to 711 W. The production run sustained 61–63K tokens/s, 3.6× the start.

The agent was tireless and careful, optimizing the variables and the metric it had chosen. The researcher contributed two judgments: the topology was not the problem, and utilization was the wrong metric. Those two sentences were worth more than five hours of search. The speedup matters beyond this run. At 63K tokens/s, mid-training is an overnight job, so we can rerun it whenever the threat landscape shifts (which it does on a near-daily basis).

Results

Every model went through the same evaluation protocol with three runs per benchmark. The security suite spans common cyber knowledge benchmarks: CTI-Bench, SecEval, SECURE, MMLU Computer Security, PrimeVul and AthenaBench.

Figure 1 shows where the gains come from. The largest are on tasks where the model has to produce structured security reasoning, not just pick an answer. In the figure, these are the rows where the green dot pulls furthest ahead:

These are the relationships the MiST data spells out, and the tasks an analyst actually hands to a model. Multiple-choice knowledge, already strong in Nemotron 3.5 Super, moves by 1–9 points. Code-level vulnerability detection (PrimeVul) is the one area where all models stay close together. In the MiST paper, we show gains there come from task-specific training on top of a MiST checkpoint, which is the job of the later stages in our stack.

A sovereign foundation

A MiST checkpoint is where the rest of Dream's national sovereign AI stack starts: task-specific fine-tuning, training on expert trajectories, and reinforcement learning with NVIDIA NeMo RL. In the MiST paper, both task-specific SFT and RL reached higher accuracy when started from a MiST checkpoint.

Every step in this post ran inside our own data center on NVIDIA DGX B200 systems: data generation, training, evaluation and serving. Open Nemotron models, NeMo AutoModel and NeMo RL, running on DGX B200, form a stack that a government can own end to end. Nemotron-3.5-Super-MiST outperforms every 120B-class open model we tested on cyber knowledge, creating a powerful model for downstream cyber tasks.

We thank NVIDIA's Nemotron and NeMo teams for early access to Nemotron 3.5 Super and for their support throughout this work.