Engineering at Meta

AUGUST 5, 2024

Distributed training, in particular, imposes the most significant strain on data center networking infrastructure. Constructing a reliable, high-performance network infrastructure capable of accommodating this burgeoning demand necessitates a reevaluation of data center network design.

Network 132

Network Networking Topology Data Centers 132

Engineering at Meta

JUNE 12, 2024

Meta is currently operating many data centers with GPU training clusters across the world. Our data centers are the backbone of our operations, meticulously designed to support the scaling demands of compute and storage. This transition has not been without its challenges.

Fashion 138

Fashion Data Centers Artificial Intelligence Server 138

Engineering at Meta

JUNE 12, 2024

This means we need to regularly checkpoint our training state and efficiently store and retrieve training data. Optimal connectivity between GPUs: Large-scale model training involves transferring vast amounts of data between GPUs in a synchronized fashion. requires revisiting trade-offs made for other types of workloads.

Infiniband 141

Infiniband Data Centers Topology Network 141

Engineering at Meta

MARCH 12, 2024

Today, we’re sharing details on two versions of our 24,576-GPU data center scale cluster at Meta. Custom designing much of our own hardware, software, and network fabrics allows us to optimize the end-to-end experience for our AI researchers while ensuring our data centers operate efficiently.

Infiniband 145

Infiniband Data Centers Server Network 145