[Remote] Databricks Data Engineer
Note: The job is a remote job and is open to candidates in USA. Leidos is seeking a Databricks Data Engineer to support the National Healthcare Safety Network support contract. The role designs, develops, and maintains scalable Databricks Lakehouse data engineering solutions, including ingestion, transformation, governance, analytics, and enterprise data pipelines. The engineer collaborates with architects, analysts, data scientists, and application teams to deliver secure, reliable data solutions for business intelligence, machine learning, and advanced analytics.
Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Apache Spark
- Build data ingestion frameworks to integrate structured, semi-structured, and unstructured data from various enterprise systems
- Develop data transformation logic using PySpark and Spark SQL
- Design, implement, and optimize Delta Lake tables for high performance, reliability, and scalability
- Implement Medallion Architecture (Bronze, Silver, Gold) for enterprise data processing
- Optimize Spark jobs through partitioning, caching, file compaction, and performance tuning techniques
- Develop and maintain data workflows using Databricks Workflows, Jobs, or orchestration tools such as Apache Airflow
- Implement data quality, validation, reconciliation, and monitoring processes to ensure data integrity
- Collaborate with cross-functional teams to understand business requirements and deliver high-quality data solutions
- Develop reusable data engineering frameworks, utilities, and coding standards
- Document data lineage, transformation logic, metadata, and technical designs
- Participate in code reviews and implement CI/CD processes using Git and DevOps tools
- Ensure compliance with enterprise data governance, security, and access management standards
Skills
- Bachelor's degree in Computer Science, Information Technology, Engineering, Mathematics, or a related discipline
- 3–7+ years of experience in Data Engineering or ETL development
- Hands-on experience with Databricks and Apache Spark
- Strong proficiency in Python and PySpark
- Advanced SQL development skills
- Experience designing and developing enterprise-scale ETL/ELT pipelines
- Experience working with Delta Lake and Lakehouse architecture
- Strong understanding of distributed data processing concepts
- Experience with data modeling, dimensional modeling, and data warehousing
- Experience working with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform
- Experience with Git-based source control and CI/CD deployment processes
- Strong analytical, troubleshooting, and problem-solving skills
- Databricks Certified Data Engineer Associate or Professional certification
- Experience with Unity Catalog, Delta Live Tables (DLT), and Databricks Workflows
- Experience with Structured Streaming and real-time data processing
Benefits
- Health and Wellness programs
- Income Protection
- Paid Leave
- Retirement
Company Overview
Company H1B Sponsorship