Lead Data Engineer - NBA

Become a part of our caring community

=========================================

The Lead Data Engineer owns the NBA platform's data foundation on the Databricks Lakehouse. This role is responsible for the architecture, delivery, quality, governance, and operational reliability of the data products that power decision intelligence, machine learning, reinforcement learning, agentic AI, and real-time decisioning across the platform. You lead a small team of engineers and contractors while remaining deeply hands-on, building and evolving the pipelines, feature layers, streaming architectures, and data quality controls that turn raw healthcare data into trusted, production-ready assets consumed across the NBA ecosystem.Key Responsibilities

• Pod delivery — Own end-to-end delivery for the data engineering pod, including planning, execution, quality, and operational readiness.
• Lakehouse architecture — Own the design and evolution of the NBA Databricks Lakehouse, including Bronze, Silver, and Gold layer standards, data lineage, and governance patterns.
• Data pipeline engineering — Design, build, and maintain scalable batch and real-time pipelines that process member, clinical, claims, behavioral, engagement, and socioeconomic datasets.
• Feature platform ownership — Lead the design and operation of Gold-layer feature tables and reusable data products that support model training, scoring, reinforcement learning, and decision intelligence workloads.
• Streaming and event architecture — Design and implement Kafka- and Spark Structured Streaming-based ingestion and processing patterns that enable near real-time decision-making.
• Data quality and observability — Establish platform-wide standards for data validation, monitoring, lineage, reconciliation, alerting, and operational visibility; treat data quality issues as production incidents.
• Performance and optimization — Drive optimization of Spark workloads, De