Tue.Dec 05, 2023

article thumbnail

API-First Approach to Kafka Topic Creation

DoorDash Engineering

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. This has reduced real-time pipeline onboarding time by 95% and saved countless developer hours. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

article thumbnail

Announcing Databricks Middle East Expansion and Launch of Azure Qatar

databricks

We’re excited to announce the launch of Azure Qatar. With the expanded availability of Azure Databricks, it is now easier than ever for o.

86
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Atlantis Hardening and Review Fatigue

DoorDash Engineering

Many organizations use infrastructure-as-code (IaC) with pull request (PR) automation to provide a more secure, safe environment for making infrastructure changes. Despite the power and flexibility of IaC software, the lack of strong, secure defaults in PR automation software can make that sense of security a false one. Infrastructure-as-code and pull request automation IaC enables a declarative, reusable, and auditable way to manage configuration changes.

article thumbnail

Managing Recalls with Barcode Traceability on the Delta Lake

databricks

Recent data show that the number of recall campaigns caused by product deficiencies keeps increasing, while each known recorded case is a multi-million.

85
article thumbnail

Where Data Governance Should Live

TDAN

Determining the optimal administrative placement for a data governance program, and specifically a Non-Invasive Data Governance program, is a pivotal decision that can significantly influence its success. This critical choice often boils down to three primary areas within an organization: a specific business area, information technology (IT), or a shared services part of the organization.

article thumbnail

Delivering Real-Time Manufacturing Predictive Maintenance

Confluent

Confluent Cloud enables organizations to unlock real-time visibility into manufacturing processes, using real-time data collection and analytics to prevent re-work and tooling failures, delivering an outsized impact on production volume and quality.

Cloud 75
article thumbnail

Article: Navigating Complex Interpersonal Relationships: Co-Creating Deliberate Workplace Connection

InfoQ Articles

As an employee or leader, you are in a relationship with each other and with the organization. You face the tension of traditional ways of working vs new ways of working in the call for more work-from-home policies, and more work/life balance. This article outlines the mindset and actions that can be taken to better navigate complex relationships leading to a more humanistic way of working.

65

More Trending

article thumbnail

Through the Looking Glass: The Unique Identifier of the Rose

TDAN

I recently taught an online class on BCBS 239: Effective Risk Data Aggregation and Reporting for Risk.net. Preparing the course materials took me back to 2007-2008, when I worked for Merrill Lynch managing the Credit Risk Reporting team.

Banking 59
article thumbnail

HS060: Power Micro-Generation for Data Center

Packet Pushers

Powering data centres is big deal in current decade. Massive increases in consumption and scaling of off-prem clouds has exceeded the capacity of civilian power grids while cloud operators are reluctant to sign thirty year supply agreements so that more power plants can be built. Enter power micro-generation where too large DCs needs too small. Read more » Powering data centres is big deal in current decade.

article thumbnail

Leveraging ISP Network Monitoring for Competitive Edge

Obkio

Unlock ISP Network Monitoring's transformative power for proactive issue detection and seamless client support, reshaping the competitive landscape.

Network 52
article thumbnail

HN712: FortiGuard Security Services: Invisible Operations, Tangible Results (Sponsored)

Packet Pushers

Today we’re talking security, but security you dont always see. Fortinet, today’s sponsor, has millions of devices in the field. These are real-world devices seeing real-world traffic, all day, everyday. While those devices have a primary protection role, they can also serve as sensors that collect threat signals and feed threat intelligence services that can, Read more » Today we’re talking security, but security you dont always see.

52
article thumbnail

The Currency of Information: The Business Value of Data (Part One)

TDAN

Data professionals often talk about the importance of managing data and information as organizational assets, but what does this mean?[i] What is the actual business value of data and information? How can this value be measured? How do we manage data and information as assets?

article thumbnail

A Data Governance Ontology

TDAN

In the era where data powers digital transformation and informs the growing number of data products, it is critical that data practitioners have a common understanding of the things that make up a data governance program. I have found recently that even standard bodies lack a common lexicon for describing data governance artifacts.

article thumbnail

Metadata Governance: An Outline for Success (Part Two)

TDAN

In part one of “Metadata Governance: An Outline for Success,” I discussed the steps required to implement a successful data governance environment, what data to gather to populate the environment, and how to gather the data.

article thumbnail

Data Is Risky Business: Turning Abstract Ethics into Practical Practices

TDAN

This quarter’s column draws on my keynote for DAMA Calgary’s contribution to DAMA Days Canada last month, which in turn drew on some of the content in the second edition of the “Data Ethics” book I wrote with my colleague Katherine O’Keefe (particularly, Chapter 3 and Chapter 11).