Databricks Data Engineer and Scientist
About the position As a Databricks Data Engineer and Scientist, you will design, build, and maintain robust data pipelines for a critical enterprise Lakehouse, applying data science, AI, and analytics skills. You will develop and optimize end-to-end data flows, manage ETL processes, design data models, help ensure data security and compliance, collaborate with cross-functional teams, and improve performance and scalability. The role uses Databricks and AWS, along with SQL, Python, and AI-assisted coding, to analyze pipelines, support data modeling, and turn raw data into actionable business insights. The position is hybrid and based in Bethesda, Maryland, with an additional location in Denver, Colorado. Applicants selected will be subject to a government investigation and may need to meet eligibility requirements of the U.S. government client. The role requires the ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements.
Responsibilities
- Design, develop, deploy, and manage ETL processes and data pipelines, including comprehensive logging and error handling.
- Build, debug, and optimize Databricks data pipelines using notebooks, clusters, job orchestration tools, Delta Lake, and Unity Catalog.
- Write Python and PySpark code for ETL scripts and custom transformations.
- Perform data modeling, selecting dimensional or relational models within a Bronze, Silver, and Gold medallion architecture.
- Use AWS services, including AWS Glue, Amazon S3, and Amazon RDS.
- Apply ETL performance optimization techniques and troubleshoot query performance.
- Analyze complex legacy SQL, rebuild data pipelines or models, and compare results to validate accuracy and performance.
- Collaborate with cross-functional teams to support data security, compliance, performance, scalability, and business insights.
Requirements
- 8+ years of experience designing, developing, deploying, and managing ETL processes and data pipelines, with comprehensive logging and error handling to troubleshoot and resolve issues.
- 3+ years of experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools, including Delta Lake and Unity Catalog.
- 3+ years of experience writing Python and PySpark code for ETL scripts and custom transformations.
- Experience using AI-assisted coding tools such as Claude Code or Cursor to support data engineering and data modeling tasks.
- Experience with data modeling, including selecting dimensional or relational models, within a Bronze, Silver, and Gold medallion architecture.
- Experience using AWS services, including AWS Glue, Amazon S3, and Amazon RDS.
- Knowledge of ETL performance optimization techniques, such as partitioning, bucketing, and caching, and experience troubleshooting query performance.
- Ability to analyze and decompose complex legacy SQL, rebuild data pipelines or models, and compare results to validate accuracy and performance.
- Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements.
- Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Math, Physics, or Information Systems and Technology.
Nice-to-haves
- Experience with AWS cloud-native data services such as AWS Lambda or Amazon CloudWatch that support serverless data processing.
- Experience with metadata and data cataloging tools such as AWS Glue Data Catalog or Collibra to manage data assets and lineage.
- Knowledge of data governance, data lineage, and compliance requirements such as GDPR and CCPA that support data quality, privacy, and security.
- Ability to explain complex technical concepts to non-technical audiences, including architectural tradeoffs.
Benefits
- Health benefits
- Life benefits
- Disability benefits
- Financial benefits
- Retirement benefits
- Paid leave
- Professional development
- Tuition assistance
- Work-life programs
- Dependent care
- Recognition awards program