[Remote] Lead Data Engineer
Note: The job is a remote job and is open to candidates in USA. Covetus is seeking a highly experienced Data Engineering Lead with expertise in Databricks, PySpark, Talend, AWS, and Big Data technologies. The ideal candidate will design and implement scalable cloud data platforms and enterprise ETL solutions, with a focus on healthcare data integration and cloud migration.
Responsibilities
- Design, develop, and optimize scalable data pipelines using Databricks, PySpark, Talend, and Snowflake
- Build and maintain cloud-based Lakehouse architecture using Databricks, Delta Lake, and AWS
- Develop ETL/ELT solutions to ingest, transform, and load data from databases, APIs, Salesforce, S3, and flat files
- Implement and manage Delta Live Tables, Databricks Workflows, and Unity Catalog for data governance
- Develop and optimize large-scale Spark and PySpark workloads for batch and real-time data processing
- Support healthcare data integration initiatives including EDI X12 transactions (834, 835, 837) and HIPAA-compliant processing
- Perform data quality validation, performance tuning, troubleshooting, and root cause analysis
- Collaborate with business, QA, and development teams to gather requirements and deliver scalable data solutions
- Support CI/CD, deployment automation, monitoring, and production support activities
Skills
- Strong hands-on experience with Databricks, PySpark, Spark SQL, and Delta Lake
- Extensive experience with Snowflake, including data loading, data modeling, and performance optimization
- Strong ETL development experience using Talend and/or IBM DataStage
- Advanced SQL expertise with strong query tuning and troubleshooting skills
- Experience with AWS services such as S3, Lambda, EMR, and cloud-based data platforms
- Proficiency in Python for data engineering, automation, and API integrations
- Strong knowledge of Data Warehousing, Data Lake, and Lakehouse architecture
- Excellent analytical, problem-solving, and communication skills
- Experience with Unity Catalog, Delta Live Tables, and Databricks governance frameworks
- Healthcare domain experience including Claims, Membership, Eligibility, and EDI processing
- Experience with Hadoop ecosystem technologies such as Hive, Kafka, HBase, Sqoop, and Oozie
- Knowledge of CI/CD tools such as Jenkins, Git, and Terraform
Benefits
- Location - 100% Remote
Company Overview
Company H1B Sponsorship