[Remote] Databricks Engineer
Note: The job is a remote job and is open to candidates in USA. Insight Global is seeking a Databricks Engineer for a remote contract role supporting enterprise data engineering initiatives. The role focuses on designing, modernizing, deploying, and supporting Azure Databricks data pipelines, while improving data quality, scalability, monitoring, and operational reliability for Medicare and payer-focused datasets.
Responsibilities
- Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables
- Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions
- Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns
- Develop and maintain CI/CD deployment processes for data engineering workflows
- Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers
- Partner with upstream and downstream teams to ensure data quality, reliability, and scalability
- Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments
- Enhance monitoring, observability, and operational support processes
- Support Medicare and payer-focused initiatives involving member, claims, and operational datasets
- Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability
Skills
- 7+ years Data Engineering for enterprise companies (preferably healthcare/payer organizations)
- Advanced/expert level Azure Databricks experience
- Advanced/expert level PySpark experience
- Delta Lake
- Delta Live Tables (DLT)
- ADLS Gen2
- Strong SQL background
- Medallion Architecture
- CI/CD and DevOps
- Configuration driven pipelines (YAML/JSON)
- Production support experience
Benefits
- 100% Remote in the US - CST/EST
Company Overview
Company H1B Sponsorship