[Remote] Lead Data Engineer
Note: The job is a remote job and is open to candidates in USA. Acumenz Consulting is seeking a Lead Data Engineer to support the migration from Palantir Foundry and on-premises MS SQL Server to Azure Databricks. The role is responsible for platform operations, performance, security, governance, automation, troubleshooting, and collaboration with data engineering teams across the existing and target data platforms.
Responsibilities
- Support and maintain Azure Databricks, Palantir Foundry, and MS SQL Server data platforms
- Drive migration of data workloads from Palantir Foundry/on-prem SQL Server to Azure Databricks
- Manage Databricks platform operations, clusters, jobs, performance, and cost optimization
- Implement Unity Catalog for data security, access, governance, and lineage
- Troubleshoot complex platform issues and perform root cause analysis (RCA)
- Develop automation using Terraform, Python, Shell, and CI/CD pipelines
- Monitor platform reliability, scalability, utilization, data quality, and governance KPIs
- Partner with data engineering teams on platform enhancements, standards, and best practices
Skills
- Need 15 years experience
- 5+ years in Data Engineering, Platform Engineering, or DevOps
- Strong hands-on Azure Databricks experience
- Palantir Foundry platform experience — preferably supporting, developing, or migrating Foundry data workloads
- Strong Apache Spark/PySpark, Delta Lake, Unity Catalog, and SQL
- Azure: ADLS, AAD/Entra ID, networking
- Terraform, Python, Shell scripting, CI/CD
- Experience with cloud data platform operations, performance optimization, security, and governance
- Experience migrating workloads from Palantir Foundry/MS SQL to Databricks is highly preferred
Benefits
- Remote work
Company Overview