[Remote] Databricks Data Engineer
Note: The job is a remote job and is open to candidates in USA. Capgemini is seeking a Databricks Engineer to lead the design and implementation of a scalable Sales Data Platform as part of the OneData initiative on Databricks running on AWS. The role involves hands-on architecture-driven tasks focused on data ingestion, transformation, modeling, and optimization using modern lakehouse patterns.
Responsibilities
- Define end-to-end lakehouse architecture on Databricks (AWS) for Sales data domains
- Design medallion architecture (Bronze / Silver / Gold) aligned with OneData standards
- Establish data modeling standards for Sales facts, dimensions, hierarchies, and aggregations
- Define scalable ingestion patterns for batch and incremental loads
- Drive performance, scalability, and cost optimization best practices
- Build and guide development of PySpark-based data pipelines in Databricks
- Implement Delta Lake features:
- ACID transactions
- Schema evolution & enforcement
- Time travel & versioning
- Design and optimize large-scale joins, aggregations, and window functions
- Implement CDC and incremental processing using watermarking and change detection
- Ensure idempotent, restartable, and fault-tolerant pipelines
- Architect solutions using AWS services:
- Amazon S3 (data lake storage)
- IAM (security & access control)
- AWS Glue / Glue Catalog
- CloudWatch (monitoring & logging)
- Optimize Databricks cluster configurations (job vs all-purpose clusters)
- Implement secrets management and secure connectivity patterns
- Define and implement data quality checks and validations
- Ensure data lineage and metadata capture
- Implement error handling, auditing, and reconciliation frameworks
- Support data governance and access control requirements
- Implement CI/CD pipelines for Databricks notebooks and jobs
- Enforce code versioning, reviews, and deployment standards
- Design monitoring, alerting, and SLA tracking for pipelines
- Support production stabilization and performance tuning
- Act as technical lead / mentor for Databricks data engineers
- Collaborate with:
- Source system teams (Sales, CRM, ERP)
- Data consumers (analytics, downstream apps)
- Cloud/platform teams
- Translate business requirements into robust technical designs
Skills
- 8+ years in Data Engineering / Data Architecture roles
- 4+ years hands-on experience with Databricks
- Strong expertise in PySpark & Spark SQL
- Strong expertise in dbt
- Deep experience with Delta Lake
- Strong knowledge of AWS cloud services (S3, IAM, Glue, CloudWatch)
- Sales data domain experience (orders, revenue, pricing, customers, products)
- Strong understanding of Fact & dimension modeling
- Strong understanding of Slowly Changing Dimensions (SCD Type 1 / 2)
- Strong understanding of Large-scale data processing patterns
- Experience handling high-volume, high-velocity datasets
- Databricks job orchestration and scheduling
- Cluster sizing and performance tuning
- CI/CD for data platforms
- Strong troubleshooting and debugging skills
- Experience with enterprise OneData / Data Mesh programs
- Exposure to real-time or near-real-time ingestion patterns
- Experience integrating CRM / Sales systems (e.g., Salesforce, SAP Sales data)
- AWS certifications or Databricks certifications
Benefits
- Medical
- Dental
- Vision
- Retirement benefits
Company Overview
Company H1B Sponsorship