Engineering at Meta

JUNE 12, 2024

This means we need to regularly checkpoint our training state and efficiently store and retrieve training data. Optimal connectivity between GPUs: Large-scale model training involves transferring vast amounts of data between GPUs in a synchronized fashion. requires revisiting trade-offs made for other types of workloads.

Infiniband 119

Infiniband Data Centers Topology Network 119

Engineering at Meta

MARCH 12, 2024

Today, we’re sharing details on two versions of our 24,576-GPU data center scale cluster at Meta. Custom designing much of our own hardware, software, and network fabrics allows us to optimize the end-to-end experience for our AI researchers while ensuring our data centers operate efficiently.

Infiniband 145

Infiniband Data Centers Server Network 145

Engineering at Meta

AUGUST 5, 2024

Distributed training, in particular, imposes the most significant strain on data center networking infrastructure. Constructing a reliable, high-performance network infrastructure capable of accommodating this burgeoning demand necessitates a reevaluation of data center network design.

Network 124

Network Networking Topology Data Centers 124