Sr Azure Databricks Engineer
Core requirement: Strong data engineering fundamentals with proven, hands-on Databricks experience. Deep experience of Databricks concepts the team uses day to day. Technical must-haves: Performance tuning and optimization, including Spark internals, join strategies, and data skew handling; strong use of LDP APIs, CDF, and CDC; and experience building data-quality frameworks using expectations, quarantine logic, and alerting. Governance & orchestration: Hands-on experience with PII/PHI masking, access controls, compliance/governance, plus the ability to orchestrate and stitch Databricks pipelines together end to end. Location: Remote- US Required Skills & Experience 5-7+ years of Data Engineering experience building large-scale data platforms
- Strong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS)
- Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning
- Experience developing and maintaining Delta Live Tables (DLT) pipelines
- Strong SQL background with experience supporting legacy data warehouse environments
- Experience migrating SQL-based pipelines to modern Python/PySpark architectures
- Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies
- Proficiency with YAML, JSON, and configuration-driven development patterns
- Experience automating CI/CD deployments using DevOps pipelines
- Strong understanding of Medallion Architecture (Bronze, Silver, Gold)
- Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers
- Experience troubleshooting, monitoring, and supporting production data pipelines
- Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferred
- Healthcare industry experience preferred
- Medicare, payer, member enrollment, or claims data experience strongly preferred Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred
Job Description
Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables
- Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions
- Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns
- Develop and maintain CI/CD deployment processes for data engineering workflows
- Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers
- Partner with upstream and downstream teams to ensure data quality, reliability, and scalability
- Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments
- Enhance monitoring, observability, and operational support processes
- Support Medicare and payer-focused initiatives involving member, claims, and operational datasets
- Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability