Engineering at Meta

SEPTEMBER 7, 2023

Arcadia gives Meta’s researchers and engineers valuable insights into the performance of AI models and workloads in an AI cluster – enabling data-driven decision making in the design of AI clusters. We’re also investigating a framework to provide design insights for different topology/routing designs given a set of known models.

Topology 112

Topology Network Networking Application 112

Engineering at Meta

SEPTEMBER 7, 2023

However, traditional full workload benchmarking presents several challenges: Difficulty in forecasting future system performance : When designing an AI system, engineers frequently face the challenge of predicting the performance of future systems. Our visualization tool can precisely highlight these imbalances, as shown by the below figure.

Network 109

Network Networking Topology Bandwidth 109

Kentik

MARCH 13, 2023

Why do we need to create site-to-site VPNs or some sort of modern SD-WAN topology connecting all our branches when almost all traffic goes to the public internet and the cloud? So what does this mean for today’s enterprise network engineer? This also means that today’s enterprise WAN poses new challenges for a network engineer.

WAN 98

WAN Wide Area Network Topology Internet 98

Packet Pushers

MARCH 20, 2024

Roman describes containerlab as a lab as code tool that quickly and easily creates virtual networking topologies. With increased automation and containerization in network engineering, the tools popularity has exploded. Roman describes containerlab as a lab as code tool that quickly and easily creates virtual networking topologies.

Topology 52

Topology Engineering Network Networking 52

Engineering at Meta

JUNE 12, 2024

We implemented collective communication patterns with network topology awareness so that they can be less latency-sensitive. The post How Meta trains large language models at scale appeared first on Engineering at Meta.

Infiniband 141

Infiniband Data Centers Topology Network 141

InfoQ Articles

JUNE 21, 2022

This article summarizes how we see the "cloud computing and DevOps" space in 2022, which focuses on fundamental infrastructure and operational patterns, the realization of patterns in technology frameworks, and the design processes and skills that a software architect or engineer must cultivate.

DevOps 145

DevOps Cloud Engineering Topology 145

InfoQ Articles

JULY 19, 2021

This article summarizes how we see the "cloud computing and DevOps" space in 2021, which focuses on fundamental infrastructure and operational patterns, the realization of patterns in technology frameworks, and the design processes and skills that a software architect or engineer must cultivate.

DevOps 145

DevOps Cloud Engineering Topology 145

Kentik

JUNE 27, 2023

Live traffic flow arrows demonstrate how Azure Express Routes, Firewalls, Load Balancers, Application Gateways, and VWANs connect in the Kentik Map, which updates dynamically as topology changes for effortless architecture reference. It also provides custom alerts and synthetic testing for each environment, including Azure.

Cloud 105

Cloud Firewall Topology Data Centers 105

Confluent

MARCH 20, 2019

One neat thing about the implementation of Suppress as an operator, rather than an option on the count or window definition, is that both the raw and final streams are available in your topology. If you have a suppression in your topology, and you write some test data, no amount of waiting will make the results come out.

Topology 106

Topology Application Email Government 106

Engineering at Meta

MARCH 12, 2024

Among other benefits, Hammerspace enables engineers to perform interactive debugging for jobs using thousands of GPUs as code changes are immediately accessible to all nodes within the environment. The post Building Meta’s GenAI Infrastructure appeared first on Engineering at Meta.

Infiniband 145

Infiniband Data Centers Server Network 145

Confluent

AUGUST 1, 2019

At the core of each shared state microservice we built was a Kafka Streams instance with a rather simple processing topology. Additionally, our processing topology includes a second key-value store for scheduling. Nitzan Gilkis is a senior software engineer at Imperva, where he works on the DDoS Protection for Networks product.

Topology 20

Topology Port Networking Network 20

Kentik

APRIL 4, 2023

More than anything, reliability becomes the principal challenge for network engineers working in and with the cloud. Even the most detailed reliability engineering can be easily undermined in an insecure network. While there is much to be said about cloud costs and performance , I want to focus this article primarily on reliability.

Cloud 104

Cloud Network Networking Bandwidth 104

Network Engineering

JULY 18, 2020

I got a router on a stick and I want to optimize it by reducing its cost and also the configuration costs This topology is to facilitate inter Vlan routing so Vlan1 can communicate with Vlan2. The question is how will this topology be optimized. The topology is illustrated in the figure

Topology 52

Topology Routers Network Networking 52

Confluent

JULY 11, 2019

The production deployment lectures allow you to confidently scale a cluster, visualize a topology and demonstrate resilience in a multi-server configuration. A data engineer architect from Sydney, Australia, he lives with his wife, two kids, and a grumpy cat. The course consists of 33 lectures in total.

Insurance 85

Insurance Topology IoT Application 85

Engineering at Meta

JUNE 12, 2024

Meta runs different types of backend networks, topologies, and training jobs that have tight dependencies between software and hardware components. The post Maintaining large-scale AI capacity at Meta appeared first on Engineering at Meta. This transition has not been without its challenges.

Fashion 138

Fashion Data Centers Artificial Intelligence Server 138

Kentik

MARCH 29, 2023

By quickly adapting to network topology changes, IGPs contribute to network resilience by rerouting traffic when a link or node fails. As a modern source-routing technique, SR simplifies traffic engineering, optimizes resource utilization, and provides better scalability than traditional routing methods.

Network 97

Network Networking Protocol Gateway 97

Network Engineering

DECEMBER 4, 2024

I want to ping from a computer located in SITE C to computers in SITE A and SITE B within the topology. One of my biggest challenges here is determining how to establish an overlay network over the WAN. Another challenge is that the network infrastructures in SITE A and SITE B share the same IP blocks, and I cannot change them.

VPN 52

VPN IP Address WAN Routers 52

NW Kings

JANUARY 7, 2025

This feature in networks predicts and stabilizes the topology. NOTE : Join our Network Engineer Master’s Program today! The disadvantages of static routing are as follows- Manual updates required for changes in topology : If a network change occurs, administrators must manually update static configurations. 255.255.255.0

Routers 52

Routers IP Address Protocol Topology 52

Kentik

MARCH 13, 2018

With thousands of servers interconnected via a leaf-spine switching architecture and traffic flowing predominantly east-west, the resulting network topologies are so complex that most operators are employing BGP routing within the data center and equal-cost multi-path routing (ECMP) to select the optimal path for each flow.

Data Centers 40

Data Centers Topology Server Artificial Intelligence 40

Network Engineering

JANUARY 8, 2025

Attached is a topology diagram illustrating the proposed setup. With proper routing, this could enable direct communication between the iPerf3 client (on the CPE or customer-side network) and our iPerf3 server, without requiring a physical technician presence.

Internet 52

Internet Bandwidth Topology Server 52

Kentik

JANUARY 29, 2020

SNMP data is also used by network engineers to troubleshoot reported problems along with network architects to do things like capacity planning. The flow analytics are used to make decisions on how traffic is being sent or received to other internet-connected peers via traffic engineering and optimization.

Port 16

Port Network Networking Data Centers 16

Kentik

JUNE 7, 2023

Kentik Cloud users can now access the new Kentik Map for Google Cloud to automatically visualize detailed Google Cloud and hybrid cloud infrastructure topology. Advanced analysis : With Kentik’s powerful analytics engine, you can perform an in-depth analysis of flow logs from any cloud.

Cloud 97

Cloud Gateway Port IP Address 97

Apium Academy

MARCH 15, 2023

Nates presentations cover a variety of topics, from software architecture, microservices, cloud computing, and site reliability engineering, to everything in between. About the Experts Michael Keeling is a software engineer at Kiavi and the author of “Design It!: George Fairbanks is a software engineer at Google.

Topology 52

Topology Engineering Education Cloud 52

Kentik

MAY 30, 2023

This includes accurate and comprehensive data on network topology, traffic patterns, historical data, and equipment specifications. No software or data engineer can do this, so a collaborative team of developers, network engineers, and planners need to work together to design and eventually use the system.

Network 52

Network Networking Topology Artificial Intelligence 52

Kentik

JANUARY 29, 2020

SNMP data is also used by network engineers to troubleshoot reported problems along with network architects to do things like capacity planning. The flow analytics are used to make decisions on how traffic is being sent or received to other internet-connected peers via traffic engineering and optimization.

Port 78

Port Network Networking Data Centers 78

Kentik

AUGUST 30, 2021

The term has a literal engineering definition, that, in a nutshell, means the internal state of any system is knowable solely by external observation. The concept of observability has taken hold in the DevOps, SRE and application performance monitoring (APM) space. Such a system is said to be “observable.”.

Network 85

Network Networking DevOps Cloud 85

Network Engineering

JANUARY 24, 2025

Let's say, that there is a following network topology: As seen on the drawing, the router named as30-r1 is connected to IXP LAN with network 10.81.80.0/22 Accepting these routes may create a black hole for connectivity to the IXP LAN. 22 while receiving and accepting 10.81.83.0/24 24 prefix from transit provider AS 20.

LAN 52

LAN Routers Topology Internet 52

Confluent

JUNE 11, 2019

It is also important to understand some of the common streaming topologies that streaming developers use to build an event streaming pipeline. Here in part 4 of the Spring for Apache Kafka Deep Dive blog series, we will cover: Common event streaming topology patterns supported in Spring Cloud Data Flow. RELEASE.jar. Named destinations.

Application 86

Application Cloud Topology Server 86

Kentik

MARCH 15, 2023

Cyberthreat strategies have evolved in step with modern cloud networks, often using cheap, virtualized cloud resources to exploit the threat surface topology I briefly described above. From a networking perspective, a lot of valuable prevention data is available to engineers who know how to use it. of automated password attacks.

Network 94

Network Networking Protocol IP Address 94

Network Engineering

NOVEMBER 13, 2015

Topology is basically Server Side (Agg2)Trust/Untrust(Agg1) Remote Office (Yamaha) 100.66.xx.0/24 I am dealing with a problem with Netscreen VPN. 0/24 Netscreen VPN Internet 100.71.YY.0/24 0/24 Proxy id is the same with source and destination range. Tunnel is in Untrust zone and unnumbered to aggregate1.

VPN 52

VPN Topology Server Internet 52

Indeed Engineering

JULY 3, 2024

Goal This blog is for engineering teams, architects, and leaders responsible for defining and implementing a workload identity platform and access controls rooted in Zero Trust principles to mitigate the risks from compromised services. SPIRE is open source, widely used in the industry and has a vibrant and active community of engineers.

Server 52

Server Topology Encryption Cloud 52

Kentik

FEBRUARY 4, 2021

Network primitives : Prefix, path, topology, and underlay and overlay are all concepts that are unique or have unique context in the network world. Without the ability to easily enrich, group, and query by prefix, network path, and under/overlay, it’s difficult for networkers to ask questions and reason about their infrastructure.

Network 80

Network Networking Topology Application 80

High Scalability

DECEMBER 18, 2018

Senior DevOps Engineer : Your engineering work will focus on using your deep knowledge of the web stack including firewalls, web applications, caches and data stores ­ to create innovative infrastructure architectures that are resilient, scalable, and blazingly fast. Please apply here. Make your job search O (1), not O ( n ).

Advertising 49

Advertising DevOps Engineering Healthcare 49

Confluent

JUNE 7, 2019

The Confluent Platform is an amazing toolbox, which every architect and data engineer should know of and utilize. Jendrik spent the last five years as a data engineer in the ad tech, peer-to-peer video streaming, and IoT industries. Why does on-chain data matter? He’s familiar with batch processing tasks in Spark and Flink.

Blockchain 93

Blockchain Real Estate IoT Financial 93

Kentik

DECEMBER 18, 2019

Last month at DENOG11 in Germany, Kentik Site Reliability Engineer Costas Drogos talked about the SRE team’s journey during the last four years of growing Kentik’s infrastructure to support thousands of BGP sessions with customer devices on Kentik’s multi-tenant SaaS (cloud) platform. IPv6 peerings are starting to outgrow a single node.

Server 13

Server Topology TCP Network 13

Rule 11

JUNE 21, 2024

Dozens of policing agencies are currently using cell-site simulators (CSS) by Jacobs Technology and its Engineering Integration Group (EIG), according to newly-available documents on how that company provides CSS capabilities to local law enforcement. Once a nonce is reused, AES-GCM completely falls apart.

Energy 40

Energy Routers Topology DNS 40

DoorDash Engineering

JANUARY 16, 2024

The cost increase prompted our engineering team to investigate alternative ways to provide the same level of service more efficiently. Enable the topology-aware routing feature for internal infrastructure traffic using the Kubernetes network proxy kube-proxy.

Cloud 84

Cloud IP Address DNS Routers 84