[Remote] Senior Data Engineer (Databricks)
Note: The job is a remote job and is open to candidates in USA. Dice is seeking a Senior Data Engineer to serve as the hands-on technical owner for its next-generation Enterprise Reporting & Data Warehouse on Databricks. The role will design and build a lakehouse architecture, migrate legacy SQL Server and Synapse systems, implement governance and AI-enabled analytics capabilities, and collaborate with architects, business users, Power BI teams, and offshore delivery partners.
Responsibilities
- Architecture & Strategy: Design the ERDW.Next lakehouse (medallion/gold layer star schema) and govern data modeling, batch/streaming, and Delta Live Tables standards
- Build & Migration: Write production-grade PySpark, Spark SQL, and Delta Live Tables pipelines orchestrated via Databricks Workflows. Re-engineer legacy SSIS/T-SQL/Synapse logic and ingest ERP data (JD Edwards, Anaplan) using CDC and Auto Loader
- Platform Operations: Optimize cost/performance using Photon, Liquid Clustering, and OPTIMIZE. Manage CI/CD pipelines via Git and Databricks Asset Bundles while maintaining strict SLAs
- Governance & AI: Implement Unity Catalog governance (entitlements, lineage, audit) and curate Databricks Genie Agents for natural-language analytics
- Partner & Legacy Management: Support legacy platforms during transition and maintain code quality standards across offshore partner teams. Leverage agentic AI coding tools safely to accelerate delivery
Skills
- * Serving as a Databricks SME
- * ERP integrations (strong plus)
- * Python / PySpark
- * Performance tuning and optimization
- * Databricks maintenance, including VACUUM
- * Data governance and best practices
- * Semantic modeling
- * Education: Bachelor's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience
- * Track Record: 5+ years dedicated to Databricks/Apache Spark platforms and 8+ years overall in Data Engineering/BI. Proven experience taking an enterprise lakehouse end-to-end into production
- * System Experience: Deep hands-on experience migrating legacy warehouses (SQL Server, Synapse, ADF) and sourcing data from ERPs (JD Edwards strongly preferred)
- * Databricks & Spark: Advanced PySpark, Spark SQL, Databricks Workflows, Delta Live Tables, Unity Catalog, Auto Loader, and Photon
- * Data Modeling & Storage: Star Schema, SCDs, Conformed Dimensions, Delta Lake (MERGE, Liquid Clustering, Z-ORDER, Time Travel)
- * Cloud & DevOps: Azure Gen2, Key Vault, Entra ID, Git, Azure DevOps/GitHub Actions, Databricks Asset Bundles, and Infrastructure as Code (Terraform)
- * Legacy & Integration: SQL/T-SQL, SSIS, SSRS, Synapse, ADF, CDC/replication tools, Direct Lake/Power BI integration, and Microsoft Fabric/OneLake
- * AI & Modern Workflow: Agentic AI coding tools, natural language query curation (Databricks Genie), and metadata-driven semantics
- * Strong technical ownership, architectural influence, and clear communication with both executives and engineers
- * Excellent cross-functional collaboration with offshore partners, citizen developers, and business stakeholders
- * Pragmatic problem-solving, critical thinking, and time management across build and legacy support
- All applicants applying for U.S. job openings must be legally authorized to work in the United States
- * Certifications (Preferred): Databricks Certified Data Engineer Professional/Data Architect; Azure DP-203 or AZ-305
Benefits
- Free online training
- Benefits are available to contract/temporary professionals, including medical, vision, dental, and life and disability insurance.
- Hired contract/temporary professionals are also eligible to enroll in our company 401(k) plan.
Company Overview