Summary: MTIA 300 combines on-chip networking and hardware-offloaded communication to significantly reduce training bottlenecks in large-scale AI recommendation models.

The newly introduced MTIA 300 AI accelerator takes a different approach to recommendation model training by prioritizing communication efficiency alongside compute performance. Designed for large-scale ranking and recommendation workloads, the chip addresses one of the most persistent challenges in AI infrastructure: the massive data exchange required between accelerators during training.

Unlike large language models, recommendation systems depend heavily on embedding tables that can account for more than 99% of total model parameters. Training these models requires continuous collective communication operations such as AllReduce, AllToAll, and AllGather across clusters of accelerators. In traditional architectures, these communication tasks often compete with AI computation for the same hardware resources, reducing overall utilization.

MTIA 300 tackles this problem by integrating networking directly into the accelerator package. The design incorporates dedicated network chiplets equipped with twelve custom 800 Gbps RDMA network interfaces, providing a total of 1.2 TB/s of I/O bandwidth. By removing dependence on external PCIe communication paths, the architecture reduces latency and improves data movement efficiency across large training clusters.

The MTIA 300 chip architecture. Diagram of MTIA 300 chip.

A key innovation is the co-designed HCCL communication framework, which treats networking as a core part of the system architecture rather than a software layer running on compute hardware. The accelerator includes 16 dedicated message engines responsible for collective communication operations, enabling AI computation and network communication to run independently.

This separation allows communication-intensive workloads to proceed with minimal impact on compute performance. The architecture delivers more than 2.8 TB/s reduction throughput, supporting high-speed execution of collective operations while maintaining near-full utilization of compute resources.

Performance results show communication bandwidth reaching 940 GB/s within a single rack. In testing with a 150-billion-parameter production recommendation model distributed across 40 accelerators, MTIA 300 reduced communication time by 3.9× compared with conventional GPU-based deployments.

A comparison of traditional accelerator design with host-based network instructions with MTIA 300’s offloaded communication model.

As AI workloads evolve toward reasoning systems, agent-based AI, and long-context inference, efficient low-latency communication is becoming increasingly important. MTIA 300 demonstrates how integrated networking and communication-focused accelerator design could shape the next generation of large-scale AI training infrastructure.