[Remote] Databricks Data Engineer

Note: The job is a remote job and is open to candidates in USA. Guidehouse is a consulting and professional services company supporting clients through its AI & Data team. The Databricks Data Engineer will design, build, and maintain scalable batch and streaming data pipelines, transform large-scale datasets, and deliver reliable data products using Databricks, Spark, SQL, Python, and cloud-native technologies. The role also involves troubleshooting, documentation, governance, and collaboration with technical teams and clients.

Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks, PySpark, SQL, Delta Lake, and related cloud-native data engineering tools
• Develop and support batch and streaming data workflows, including ingestion, transformation, validation, and publishing of curated data products
• Write, optimize, and maintain Python, PySpark, and SQL code for data processing, orchestration, and performance tuning
• Work with large-scale datasets using Databricks, Spark, Delta Lake, Unity Catalog, and cloud storage services
• Troubleshoot and resolve data pipeline failures, performance issues, data quality issues, and workflow bottlenecks
• Translate business and technical requirements into data engineering designs, processing scripts, and job orchestration workflows
• Perform data validation, quality checks, code reviews, and issue resolution to support accurate and reliable data products
• Collaborate with cross-functional teams including solution architects, data scientists, analysts, DevOps engineers, and client stakeholders
• Communicate technical concepts, delivery impacts, and data engineering considerations to technical and non-technical audiences
• Document pipelines, datasets, data models, workflows, and engineering decisions to support maintainability, transparency, and reuse
• Follow data governance, security, lineage, and compliance standards within the Databricks platform

Skills

• * Bachelor's degree in computer science, engineering, mathematics, statistics, or another relevant field
• * 3-8 years of relevant experience in data engineering, data architecture, or cloud data platform implementation
• * Strong experience with Python, PySpark, and SQL for data transformation, pipeline development, and data processing
• * Experience with Databricks, Spark, Delta Lake, Unity Catalog, or similar cloud-native data platforms
• * Experience developing batch or streaming data pipelines, ETL/ELT workflows, and reusable data assets
• * Experience with data modeling, data warehousing, data quality validation, and large-scale data processing concepts
• * Ability to troubleshoot technical issues, communicate engineering recommendations clearly, and work effectively in team-based delivery environments
• * Experience applying data governance, access control, lineage, and security practices within cloud data platforms
• * 2+ years of hands-on experience with the Databricks platform
• * Experience using Unity Catalog for data governance, schema design, data modeling standards, access controls, lineage, and secure management of enterprise data assets
• * Experience designing and building standard medallion data architectures for scalable ingestion, transformation, quality validation, and curated data delivery
• * Active Databricks Data Engineer Associate, Databricks Data Engineer Professional, or related certification
• * Experience with CI/CD tools and practices for notebooks, workflows, orchestration jobs, and infrastructure deployment
• * Experience with cloud platforms such as Azure, AWS, or GCP
• * Experience working in project-based or consulting delivery environments
• * Familiarity with streaming frameworks, orchestration tools, Terraform, or related data platform administration capabilities
• This position offers virtual work flexibility within the United States. While remote candidates will be considered, preference will be given to candidates located near a Guidehouse office in one of the following markets: Arlington, VA; Washington, DC; New York, NY; Chicago, IL; Austin, TX; Atlanta, GA; Boston, MA; and Boulder, CO.

Benefits

• Virtual work flexibility within the United States
• A flexible benefits package
• A comprehensive total rewards package

Company Overview

Company H1B Sponsorship