NVIDIA’s Nemotron 3.5 Lightning Is Built for the AI Grunt Work
Most new AI models arrive with the same basic sales pitch: this one is smarter. NVIDIA’s Nemotron 3.5 Lightning is interesting for almost the opposite reason. It’s designed to be fast and efficient enough to handle the enormous amount of routine work created when AI systems stop answering one question at a time and start acting as agents.
Published: August 16, 2026
Estimated reading time: 5 minutes
That means tool calls, code checks, document searches, command results, formatting, verification and the small decisions that pile up inside a larger task. The expensive model can decide what needs to happen. Lightning can handle much of the execution.
A 30-billion-parameter model that uses about 3 billion at a time
Lightning is a 30-billion-parameter mixture-of-experts model, but NVIDIA says only about 3 billion parameters are active for each token. It combines that MoE design with Mamba-2 and selected attention layers. Instead of engaging the entire model for every piece of text it generates, an MoE model routes work through a smaller selection of specialized parameters. Lightning can therefore carry the capacity of a much larger model without paying the full computational cost on every token.
Lightning isn’t being positioned as the one model that should answer every difficult question. NVIDIA describes agentic AI as a system in which larger reasoning models handle planning and harder decisions while smaller models handle narrower execution tasks.
Imagine a coding agent trying to repair a broken application. A powerful model might determine the likely cause, decide which files need inspection and plan the repair. The work that follows can involve dozens or hundreds of smaller interactions: reading files, checking command output, comparing test results and deciding whether a step succeeded. Why use the most expensive model just to check whether a command succeeded?
NVIDIA also introduced NeMo Switchyard, which is designed to route different requests to different models. Put the two ideas together and the strategy becomes clearer: use as much model as the job requires, but no more.
Speed starts to change the economics of agents
For an ordinary chatbot, a faster answer is mostly a convenience; for an autonomous agent making thousands of calls, speed and inference cost can decide whether the product is practical at all.
NVIDIA says Lightning can generate output up to four times faster than comparable small open models. In NVIDIA’s PinchBench testing, it completed 10,000 agent tasks about 30 percent faster than Qwen3.6 35B while producing similar accuracy. Those are NVIDIA’s own benchmark results, so independent testing still matters.
The design also uses multi-token prediction and supports speculative decoding. Both techniques attack the same basic problem: generating text one token at a time can be painfully inefficient when a model is being asked to do a huge amount of repetitive work.
Thoughtworks, which tested Lightning before release, reported throughput gains of roughly 1.46 to 1.96 times over ordinary decoding on H200 and B200 GPUs without a measurable accuracy loss in its tests.
Small enough to become a specialist
Lightning also supports a context window of up to one million tokens and is available in BF16 and NVIDIA’s smaller NVFP4 format (a quantized version). NVIDIA lists deployment options ranging from data-center GPUs to a DGX Spark, and the model is available through common local-model software including vLLM, llama.cpp, Ollama and LM Studio.
That opens another possibility: companies can customize versions of Lightning for specific internal jobs instead of sending every request to one general-purpose cloud model.
A company could tune one version around its codebase, another around internal documentation and another around a narrow operational workflow. Thoughtworks reported that it post-trained specialized legal and healthcare versions in a few hours on a single node, with its legal adapter outperforming the base model in blind comparisons.
The bigger idea is the model hierarchy
Lightning probably won’t replace the biggest models from OpenAI, Anthropic, Google or NVIDIA itself. That misses the point.
The more interesting shift is away from the idea that an AI application should choose one model and use it for everything. An agent can have a powerful model making difficult decisions while faster specialists quietly perform the repetitive work underneath it.
Users may eventually interact with a single AI product while that product moves requests among several models behind the scenes.
If that architecture becomes common, the AI race changes. Building the smartest model will still matter, but so will identifying the cheapest model capable of completing each piece of work and routing the task correctly.
It looks like NVIDIA is betting that the future of AI will not be one giant brain doing everything. It will be a hierarchy of models, with the expensive ones thinking and the fast ones getting the work done at scale and at lower cost.