SemiAnalysis拆解算力荒：OpenAI与Anthropic正在抢光全球新增算力
“民主党拿下德州”加剧AI数据中心担忧，英伟达、CoreWeave盘后走低
PlantVLM-Edge: Quantized Vision-Language Models for On-Device Crop Health Diagnosis
Turbo-GMM: FlashAttention-Inspired GPU-Accelerated Gaussian Mixture Model
MatMoE: Matryoshka Mixture-of-Experts with Dynamic Mixed-Precision Quantization for Efficient Inference
LoRAForge: Co-Designing Post-Hoc Compression and Execution for Efficient Multi-LoRA Serving
From Single-Shot QA to Continuous Perception: Revisiting the Energy Composition and Optimization Boundary of Edge VLMs
The SIMPLE Law for Energy Cost of LLM Inference
Cities Worldwide Build and Scale AI Services With NVIDIA Physical and Agentic AI
Cities generate enormous amounts of data from cameras, sensors and operational systems, but turning it into useful action — for planning, maintenance and public services — remains a challenge.
Physical and agentic AI offer new ways to close that gap, from digital twins for infrastructure planning to vision and service agents, while keeping local governments in control of how sensitive data is processed and used.
Building on initiatives announced at GTC Berlin, NVIDIA and its partners are expanding physical AI and AI agents to cities worldwide. Shared computing infrastructure gives city departments, neighboring municipalities and local developers a common foundation to build on — extending services without each organization having to acquire and manage resources of their own.
Agentic AI for Cities
From resident services to vision agents, cities are adopting NVIDIA agentic AI to deliver faster, more responsive public services. City teams don’t need a massive infrastructure investment to get started building these AI services: depending on the workload, NVIDIA RTX workstations or a small GPU system may be enough.
In Spain, 1MillionBot combines open source ALIA models trained at the Barcelona Supercomputing Center with NVIDIA Nemotron and other models in resident-service agents for Madrid Digital (Comunidad de Madrid), Administració Oberta de Catalunya and 150+ deployments across Spain, handling municipal procedures, appointment requests, urban-planning incidents and service inquiries.
In the San Francisco Bay Area, LiveX is bringing its AI City vision to life with NVIDIA Cosmos world models and Nemotron open models, and Omniverse and Metropolis libraries on NVIDIA RTX PRO GPUs. The interactive, conversational AI hosts understand people’s needs, provide answers and take action, and can be tailored to local languages and service needs in airports, hospitals, local events and other public spaces.
Vision AI agents can turn existing city camera feeds into operational insights, helping teams identify mobility and safety issues and coordinate emergency responses across city services.
In Wrocław, Poland, K2K is building shared AI infrastructure using NVIDIA accelerated computing, NVIDIA Cosmos world reasoning models, and NVIDIA Metropolis Blueprint for video search and summarization (VSS) to transform existing camera streams into actionable intelligence for mobility, safety and event response — using the same infrastructure across several city functions.
In Malaysia, ITMAX bundles infrastructure and agentic AI applications into long-term managed services for cities, including for traffic management. Built with reference to Cosmos, Nemotron and the Metropolis VSS blueprint, ITMAX’s SamurAI V2 vision agents are scaling on NVIDIA RTX PRO 6000 GPUs to support 17,000+ cameras by 2027. ITMAX integrates TeraSim, an open source traffic simulation platform from Laplace Intelligence, to test signal timing, reducing peak-hour corridor travel times by up to 26%.
Plan and Operate Critical Infrastructure With Digital Twins
Beyond public-facing services, digital twins can help cities expedite infrastructure planning, minimizing disruption to local communities. City digital twins can also serve as a real-time operational simulation to facilitate faster decision-making.
At Dangote’s Lekki refinery in Nigeria, Honeywell Technologies’ Forge Plant Cognition uses Nemotron-powered AI agents to understand operational data. A digital twin of the plant and its processes, built using Omniverse libraries, is also planned to support simulation and training. The platform, built on Dell PowerEdge servers with NVIDIA RTX PRO accelerators, helps operators identify bottlenecks, anticipate equipment and process issues, and optimize plant performance.
In Taiwan, Linker Vision and Dutch architecture firm MVRDV are using NVIDIA Omniverse and Nemotron to build environmental, social and governance agents that can assess the environmental impact of various urban planning scenarios, simulating 15-year outcomes of an urban forest canopy in 10 minutes — increasing design iteration speed from two weeks per iteration to less than an hour.
Build a Broader Local AI Ecosystem
Cities can extend AI to more services by building on infrastructure already in place and adding capacity. Shared resources and managed services can help extend those capabilities across departments and municipalities.
At Morocco’s Tanger Med port, Cires Technologies is expanding a sovereign AI factory for planned applications in worker safety, congestion analysis, cargo routing and port operations. Cires Technologies is also building an ecosystem of application providers to develop local models and applications for ports and cities.
Beyond city services, Barcelona Supercomputing Center’s AI Fact
AI Is Changing How Semiconductor Fabs Work — and Engineers Are Moving Faster Because of It
Demand for advanced chips is rising. The processes to build them are getting more complex. And the margin for error is shrinking.
For engineers, understanding how a process change will behave, how a material will interact with a new step, how a defect maps back to a root cause are critical, challenging steps in semiconductor manufacturing — and these analyses can take days. Physical experiments on a fab line are slow and expensive. A wrong call on yield or process quality has real consequences.
AI is transforming semiconductor manufacturing, impacting what engineers can do — and how fast.
Accelerated computing shortens simulation time, allowing engineers to evaluate more alternatives before committing to physical trials.
Physics-based AI helps engineers predict process behavior without running a full simulation every time.
AI agents handle sequences of engineering tasks that once consumed hours.
And digital twins let teams test changes in a virtual environment before committing to physical trials.
Together, these capabilities describe an AI-defined fab: a semiconductor manufacturing factory where AI helps engineers optimize processes, catch problems earlier and make better decisions across the full manufacturing chip lifecycle. Engineers direct the work and make decisions based on their expertise while spending less time waiting for answers.
Shorter Simulation Cycles Change What Engineers Can Explore
Accelerated computing doesn’t just speed up individual calculations — it changes how many alternatives engineers can test before committing.
Multiphysics simulations model interactions among physical effects, such as heat transfer, electrical behavior and mechanical stress.
Using the NVIDIA cuDSS library, multiphysics software provider COMSOL is accelerating such simulations for semiconductor manufacturing by up to 10x. This helps engineers optimize equipment and processes, compare alternatives and decide which changes to test physically.
Applied Materials and NVIDIA are collaborating to accelerate atomic-scale materials discovery and process development. By integrating NVIDIA cuDSS, Applied’s Ginestra simulation platform runs up to 10x faster than CPU-only approaches. And with NVIDIA PhysicsNeMo, Applied’s ACE+ multiphysics platform delivers topography simulations up to 35x faster.
In some cases, acceleration doesn’t just shorten the calculation — it enables it altogether. Computational lithography — which defines how circuit patterns are printed onto silicon — involves calculations too intensive to complete on CPU at production scale. Siemens is integrating NVIDIA cuLitho into its Calibre lithography software, delivering up to 17x acceleration on key simulation workloads. That expands what engineers can verify before a wafer goes to print.
As such, faster calculations can help researchers explore material behavior and reaction pathways that influence next-generation semiconductor processes.
Predicting Process Behavior Without Running Every Simulation
In addition to accelerating simulations, AI trained on physics can predict how materials and processes will behave — without running a full simulation each time.
The payoff is most visible in complex calculations that are expensive even on accelerated hardware.
Plasma — the electrically charged gas used in semiconductor etch processes — is one example. Modeling plasma behavior requires solving equations that capture how electrical, chemical and thermal processes interact, making repeated simulation costly. Lam Research demonstrated that the NVIDIA PhysicsNeMo framework can predict plasma behavior in milliseconds, compared with days for a full simulation — giving engineers a faster way to explore process conditions without rerunning the full model each time.
Agents Handle the Steps Between Decisions
Engineers turn analyses into decisions — revising a design, investigating a test failure, adjusting a production process and more.
AI agents can help carry out the steps between those decisions.
Evaluating which AI model to apply to a given task is one example: a sequential, manual process that can consume hours on the test floor.
Advantest is using NVIDIA NeMo Switchyard AI agents to automate that selection, reducing response times from hours to minutes.
Testing Factory Changes Before They Reach the Line
Digital twins let teams simulate equipment configurations, layouts and process changes in a virtual environment before committing to physical trials. This reduces experimentation costs, especially as fabs become more complex and the stakes of a wrong change get higher.
NVIDIA Shows 3 Ways to Speed Up MoE Inference on DGX Spark, Lifting Ling-3.0-flash Throughput 42.5%
NVIDIA’s latest technical blog shows how to speed up mixture-of-experts (MoE) inference on one DGX Spark. The test model is Ant Group inclusionAI’s Ling-3.0-flash, a 124B-parameter MoE with 5.1B active parameters. NVIDIA stacked three optimizations: a new MoE kernel backend, an FP8 output layer and built-in speculative decoding. On all 1,319 GSM8K questions, decode throughput rose 42.5% with no statistically detectable accuracy change.
So why is it relevant for Devs? DGX Spark is a desktop system, not a data center node. NVIDIA’s results show a 124B MoE serving single-user requests faster on that desk-side box. The technical post also shows where MoE decode time goes, and how to cut it.
TL;DR
- Size: Ling-3.0-flash-fp4: 124B total, 5.1B active per token; 256K native context.
- Runs on: One NVIDIA DGX Spark (GB10, 128 GB unified memory, 273 GB/s), batch size one.
- Performance: Three stacked optimizations lift GSM8K decode throughput from 37.81 to 53.89 tokens/s.
- Best: +42.5% throughput and -24.5% mean latency, with accuracy flat (96.89% vs 96.82%).
- Lagging: Synthetic gains (+185.1%) shrink on GSM8K as MTP acceptance drops from 3.90 to 2.68.
Why is MoE decoding on DGX Spark memory-bound?
DGX Spark is NVIDIA’s desktop system built on the GB10 Grace Blackwell Superchip. It offers 128 GB of unified memory and 273 GB/s of memory bandwidth. Peak compute is up to 1 petaFLOP of sparse FP4. NVIDIA rates it for inference on models up to 200B parameters.
In the NVIDIA technical post, the NVIDIA team profiled Ling-3.0-flash-fp4 at batch size 1. Decoding was memory-bound: reading model weights drove execution time. Two operations stood out, the MoE computation and the LM head that scores the vocabulary. That points to two levers: move fewer bytes per decode step, and generate more tokens per verification round.
The baseline followed the official W4A8 recipe. It used FlashInfer MXFP4 MoE kernels, CUTLASS dense GEMM and an FP32 LM-head path. On a synthetic 32K-input, 2K-output test, it ran at 27.71 ms per token (36.09 tokens per second).
What are the 3 optimizations?
1. Humming MXFP4 MoE kernels: NVIDIA swapped FlashInfer’s MoE backend for Humming, an independent kernel library. Humming uses NVRTC to compile kernels at runtime for the actual matrix shapes, data types and GPU. That includes the narrow 768-wide expert layer. Each kernel compiles in seconds. The FlashInfer CUTLASS path compiles 97 targets with ninja, on the order of tens of minutes. Throughput rose 9.3% to 39.45 tokens per second.
2. Online FP8 LM head: The LM head is a 157,184 x 2,560 matrix stored in BF16. Every generated token reads all of it. Quantizing only this layer to FP8 at load time halves those reads. Setting SGLANG_ENABLE_FP8_LM_HEAD=1 enables it. It cannot be combined with the baseline –enable-fp32-lm-head option. Combined with Humming, throughput reached 43.94 tokens per second. That is 21.7% over baseline and 11.4% over Humming alone.
3. MTP speculative decoding with NEXTN: Ling-3.0-flash ships with a built-in NEXTN prediction layer, so no extra draft model is needed. It drafts three tokens, and the model verifies four positions at once. Verification costs about 2.26x a normal decode step. The main cost is MoE, not attention. Verifying four tokens can activate up to 32 experts instead of 8. On the synthetic test, an average of 3.90 tokens were accepted per round. Throughput jumped to 102.88 tokens per second, up 185.1%.
The three techniques do not overlap. Humming and the FP8 LM head cut the cost of each step. MTP cuts the number of steps.
The integration trap: Stacking these is not automatic. With FP8 LM head and MTP both on, the draft must share the full LM-head module. Otherwise the framework silently falls back, and the draft interprets FP8 weights as BF16. Accept length drops to 1.00, and throughput falls below the unoptimized baseline. The NVIDIA team advises checking at startup that each optimization is active. NVIDIA points to the Ling cookbook notebook for enabling both together.
What happens on a real workload?
Synthetic tests flatter speculative decoding, so the NVIDIA team also ran all 1,319 GSM8K questions. It used 5-shot prompting with chain-of-thought, temperature 0.6, top_p 0.95 and top_k 29. With all three optimizations, time per output token fell from 26.45 ms to 18.56 ms. Decode throughput rose 42.5%, from 37.81 to 53.89 tokens per second. Mean latency dropped 24.5%, from 15.21 to 11.49 seconds. Accuracy moved from 96.89% to 96.82%, a change NVIDIA calls statistically undetectable.
Accept length here was 2.68, versus 3.90 on synthetic data. The NVIDIA team gives three reasons for the gap. Synthetic outputs are more repetitive, so drafts are accepted more often. The synthetic test used greedy decoding, while GSM8K used sampling. GSM8K answers are also short, with a median near 310 tokens versus a fixed 2K.
&&What is Ling 3.0, and why does it suit DGX Spark?
Ling-3.0-flash belongs to Ant Group inclusionAI’s Ling 3.0 family, released under the MIT license. The open-weight lineup also includes Ling-3.0-tiny, a 7.9B MoE with 1.3B active parameters. Both come with base checkpoints and FP8, INT4 and GGUF builds; flash also has FP4. OpenRouter also lists inclusionAI’s vision-language, finance and health variants built on Ling 3.0 Flash.
Ling-3.0-flash is built to keep per-token cost low:
- Sparse MoE: Each MoE layer routes a token to 8 of 512 routed experts, along with 1 shared expert. About 4.1% of the weights are active per token.
- Hybrid linear attention: 35 Kimi Delta Attention (KDA) layers and 7 Gated MLA layers alterna
Anthropic Releases FlashProve: Cryptographic Proofs That a Reply Came From Committed Weights
Anthropic’s Frontier Red Team has released FlashProve, an open-source research prototype for provable inference. FlashProve is a zkSNARK system from Anthropic that proves a reply came from a committed model. The proof reveals nothing about the model weights. On Llama 4 Maverick, a 400B mixture-of-experts model, it proves about 110 tokens/second on 1 H100. For developers, this targets a real gap: today you cannot check which model actually answered your API call.
TL;DR
- Size: Proves a 400B MoE model (~17B active per token); context capped at 1,024 tokens per request.
- Runs on: Prover needs 1 H100 80 GB, 24 CPU cores and ~290 GB host RAM. Verifier runs on a CPU, no GPU.
- Performance: Orders of magnitude faster than prior open-source zkML systems on the same 400B model.
- Best: 110 tokens/second proving on 1 H100, with 18.78 MB proofs checked in 17 seconds.
- Worst: Proving costs ~70 to 95x the GPU time of plain vLLM inference.
What problem does provable inference solve?
A user querying a model cannot verify which model produced the response. A routing service could swap in a cheaper model. An attacker could tamper with the token stream. Judging from output quality alone is unreliable.
Anthropic lists 4 use cases:
- Regulators could confirm hardware serves inference, supporting AI pacing rules.
- Labs could detect tampered inference servers.
- Users could verify text provenance.
- Regulators could enforce serving of approved models only.
How does FlashProve work?
FlashProve has a prover (the model provider) and a verifier (the user). It runs in 2 phases:
- Offline: The provider publishes a commitment, a short fingerprint binding one exact set of weights. This takes about 39 minutes and covers about 16 million requests.
- Online: For each request, the GPU produces a proof that the committed model produced those tokens.
The proof system compiles an interactive oracle proof into a cryptographic proof using hash functions. Anthropic says this offers post-quantum security. It is the same family used by validity-proof L2 networks such as Starknet.
Three engineering choices drive the speed. Prover work scales with the ~17B active parameters, not all 400B. Requests are proved in batches after the fact, like prefill. Weights stream from host memory, so a single 80 GB GPU suffices.
How fast is it? Benchmarks
Anthropic measured the public 4-bit Maverick weights on 1 H100 with 24 CPU cores. Per the repo performance guide:
- Batch of 24: 111.23 tokens/second, ~221 to 226 seconds per batch.
- Batch of 1: 17.41 tokens/second. Batch of 64: 123.31 tokens/second, flagged as unreliable on 80 GB cards.
- Proof size: 18.78 MB. Client downloads ~6.5 MB once per commitment.
- Verification: 17 seconds on 24 threads, 21 seconds on 1 thread.
Against vLLM running FP8 Maverick on 8 H100s, proving costs about 72x the GPU time per batch. At vLLM’s highest tested load, the gap is about 94x.
&How does FlashProve compare with other zkML systems?
| Metrics | FlashProve | Attestable | VerInf | DeepProve | zkLLM |
|---|---|---|---|---|---|
| Model proven | Llama 4 Maverick (400B MoE) | Gemma 4 31B (dense) | Llama 4 Maverick (400B MoE) | GPT-2, Gemma 3 | LLaMA-2-13B |
| Throughput | ~110 tok/s | 53 to 77 tok/s | ~0.02 tok/s (1,000 tokens in 14.3 h) | ~2.9 tok/s (GPT-2), ~1.4 tok/s (Gemma 3) | Full inference in under 15 min |
| Hardware | 1 H100 + 24 CPU cores | 1 H100 | 1 DGX Spark | 1 machine; 31 tok/s distributed | GPU (CUDA) |
| Proof size | 18.78 MB | 4.35 to 7.92 MiB | 93.6 GB | Not disclosed | Under 200 kB |
| Verify time | 17 s (24 threads) | 157 to 648 ms | 17.7 h (20 cores) | 1 to 3.7 s | Not disclosed |
| Context | 1,024 tokens | Up to 16K tokens | 1,000 tokens | Not disclosed | Not disclosed |
| Code / license | Open, Apache 2.0 | Not open source | Open, MIT | Open source per Lagrange | Paper (CCS 2024) |
How is the security checked?
Cryptographic code is brittle, so Anthropic formalized the protocol in Lean. It proves completeness, soundness and zero knowledge. The README states a cheating provider is caught except with probability about q/2^124 + 2^-101.
FlashProve ships 2 verifiers. A fast C++ verifier is for daily use but carries no formal guarantee. A slower Lean verifier is proven to accept only what the soundness theorem accepts. Anthropic team notes Claude helped optimize the system.
Key Takeaways
- FlashProve proves a 400B MoE model at ~110 tokens/second on 1 H100.
- Proofs are 18.78 MB and verify on a CPU in 17 seconds.
- Proving costs ~70 to 95x the GPU time of vLLM inference.
- Security properties are formalized in Lean; weights stay hidden.
- Limited to 1,024 tokens, greedy decoding, and MoE models for now.
Check out the research post, technical report, protocol specification and GitHub repo. Als
Introducing @huggingface/kernels: 200+ WebGPU Kernels for Local AI
Welcome Le Chonk: Mistral Large 4 on Hugging Face
