High-Throughput AI Inference Chips

Microsoft’s Maia 200 is a new in-house accelerator chip built to handle large-scale AI inference workloads across its cloud. Positioned as a follow-up to the Maia 100 introduced in 2023, the chip is designed to run today’s largest AI models more quickly while using power more efficiently. With over 100 billion transistors and double-digit petaflop performance at low precision, it focuses specifically on the compute stage where trained models generate outputs.

The Maia 200 delivers over 10 petaflops of 4-bit performance and around 5 petaflops at 8-bit, representing a sizable jump over Microsoft’s earlier silicon. The company reported that individual Maia 200 nodes are already powering internal systems, including models from its Superintelligence team and Copilot services. Alongside the hardware, Microsoft has opened access to a Maia 200 software development kit, inviting developers, academic researchers, and advanced AI labs to tune workloads for the new architecture.

For AI builders, Maia 200 supports a growing shift toward custom data center chips that address inference costs, which have become a major operating expense as models scale. By offloading work from general-purpose GPUs, the chip enables more predictable capacity and potentially lower energy consumption for continuous AI services. This reinforces a broader trend in cloud infrastructure where major platforms design vertically integrated stacks, pairing proprietary hardware with specialized software to streamline AI deployment at scale.

Image Credit: Microsoft