[Remote] Data Engineer - Databricks
Note: The job is a remote job and is open to candidates in USA. Resultant is a modern consulting firm that partners with clients to solve complex challenges through data analytics and technology solutions. The Data Engineer will design, build, and optimize Lakehouse-based data platforms and collaborate with various stakeholders to deliver production-grade solutions.
Responsibilities
- Design, build, and optimize scalable ETL/ELT pipelines on Databricks using PySpark, Spark SQL, and Delta Lake
- Implement Medallion (Bronze/Silver/Gold) architecture patterns, applying data quality checks, schema evolution, and enforcement along the way
- Build declarative pipelines with Delta Live Tables (DLT) and ingest streaming/incremental data using Auto Loader and Structured Streaming
- Orchestrate and monitor production workloads using Databricks Workflows, integrating with tools like Airflow or Azure Data Factory where needed
- Configure and maintain Unity Catalog for data governance — catalogs, schemas, access controls, lineage, and PII masking
- Partner with data scientists to prepare feature-engineered, ML-ready datasets and support model deployment workflows using MLflow
- Tune cluster configuration, job design, and Photon/serverless compute for performance and cost efficiency
- Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and asset bundles (Git-based workflows, Azure DevOps, GitHub Actions, or similar)
- Query, profile, and assess the quality of large, complex datasets from a wide variety of source systems
- Collaborate with solution leads, architects, and project managers on solution design and technical architecture decisions
- Participate directly in client-facing work: requirements gathering, solution reviews, and translating technical tradeoffs into plain-language business impact
- Document solutions clearly — architecture diagrams, data flow documentation, code comments, and runbooks
Skills
- Bachelor's degree in Computer Science, Engineering, Data Science, or a related field (or equivalent practical experience)
- 2+ years of hands-on data engineering experience, including production work on the Databricks platform
- Strong hands-on experience with PySpark and Spark SQL
- Practical experience with Delta Lake fundamentals — ACID transactions, OPTIMIZE/Z-ORDER, partitioning, and schema evolution
- Solid SQL skills across relational platforms (SQL Server, Postgres, Oracle, Snowflake, etc.)
- Experience with at least one major cloud platform (Azure, AWS, or GCP) and its data services
- Working knowledge of data modeling (dimensional modeling, 3NF) and ETL/ELT design principles
- Strong communication skills and comfort working directly with clients and non-technical stakeholders
- A collaborative, detail-oriented mindset with a bias toward solution quality and follow-through
- Databricks Certified Data Engineer Associate or Professional
- Experience with Unity Catalog, Delta Live Tables, and Auto Loader in production environments
- Exposure to MLflow, Feature Store, or Databricks Vector Search for AI/ML-enabled use cases
- Experience with Databricks Asset Bundles and CI/CD tooling (GitHub Actions, Azure DevOps, GitLab)
- Familiarity with Terraform or other infrastructure-as-code tooling
- Experience with dbt, Kafka/Event Hubs, or BI tools (Power BI, Tableau) connected to Databricks
- Docker/Kubernetes experience for containerized workloads
- Prior consulting experience, or comfort moving across multiple client engagements and industries
Company Overview
Company H1B Sponsorship