Remove Application Remove Engineering Remove Topology
article thumbnail

Optimizing Kafka Streams Applications

Confluent

Kafka Streams introduced the processor topology optimization framework at the Kafka Streams DSL layer. In what follows, we provide some context around how a processor topology was generated inside Kafka Streams before 2.1, Along with it, we will demonstrate a few known issues that impact efficiency of the generated processor topology.

article thumbnail

Watch: Meta’s engineers on building network infrastructure for AI

Engineering at Meta

The 2023 edition of Networking at Scale focused on how Meta’s engineers and researchers have been designing and operating the network infrastructure over the last several years for Meta’s AI workloads, including our numerous ranking and recommendation workloads and the immense GenAI models.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

A RoCE network for distributed AI training at scale

Engineering at Meta

Topology We built a dedicated backend network specifically for distributed training. To support large language models (LLMs), we expanded the backend network towards the DC-scale, e.g., incorporating topology-awareness into the training job scheduler. We designed a two-stage Clos topology for AI racks, known as an AI Zone.

Network 132
article thumbnail

Arcadia: An end-to-end AI system performance simulator

Engineering at Meta

Arcadia gives Meta’s researchers and engineers valuable insights into the performance of AI models and workloads in an AI cluster – enabling data-driven decision making in the design of AI clusters. Our multi-layered system : At Meta, we control the stack from physical network to applications.

Topology 112
article thumbnail

Today’s Enterprise WAN Isn’t What It Used To Be

Kentik

Yes, there’s something to say about how applications are written, but on the public internet side, we’ve seen a decrease in latency, cost, and a massive increase in available bandwidth. So what does this mean for today’s enterprise network engineer? Just think about the audio quality of your last Zoom call.

WAN 98
article thumbnail

Announcing Complete Azure Observability for Kentik Cloud

Kentik

Kentik customers move workloads to (and from) multiple clouds, integrate existing hybrid applications with new cloud services, migrate to Virtual WAN to secure private network traffic, and make on-premises data and applications redundant to multiple clouds – or cloud data and applications redundant to the data center.

Cloud 105
article thumbnail

Kafka Streams’ Take on Watermarks and Triggers

Confluent

Whether Streams emits every single update or groups updates is irrelevant to the semantics of a data processing application. High-volume applications may not be able to process and transmit every update within the constraints of CPU, memory, network and disk. You get to focus on the logic of your data processing pipeline.

Topology 106