[Remote] Senior Data Engineering Support Engineer - Databricks & PySpark
Note: The job is a remote job and is open to candidates in USA. NTT DATA is recruiting for a confidential client seeking a Senior Data Engineering Support Engineer. The role supports production Databricks and PySpark data pipelines through incident response, performance and cost optimization, reliability automation, and technical mentorship.
Responsibilities
- Critical Incident & Escalation Management: Act as the highest tier of internal escalation for complex, high-priority production failures in massive-scale batch and streaming PySpark jobs
- Deep-Dive Performance Tuning: Drive the resolution of complex distributed computing bottlenecks, including severe data skew, Out-Of-Memory (OOM) errors, improper partition pruning, and costly shuffle operations using Spark UI, logs, and Ganglia metrics
- Cost & Cluster Optimization: Analyze Databricks cluster utilization and run strategies to reduce cloud compute spend (e.g., configuring auto-scaling, spot instances, and choosing optimal driver/executor node types)
- Governance & Reliability Frameworks: Oversee and troubleshoot data access control, schema evolution, and data lineage tracking within Databricks Unity Catalog
- Automation & SRE Mindset: Build robust, automated frameworks and scripts in Python/PySpark to automate data reconciliation, system health checks, and self-healing pipelines
- Strategic Playbooks & Documentation: Author comprehensive, technical runbooks and post-mortem RCA documents. Establish SOPs (Standard Operating Procedures) to reduce future incident resolution time (MTTR)
- Mentorship & Leadership: Provide technical guidance to junior support engineers, conduct technical code reviews for hotfixes, and closely collaborate with core development teams to bridge the gap between engineering and operations
Skills
- 7+ years of total experience in Data Engineering
- 3+ years dedicated to supporting or optimizing large-scale Databricks and PySpark systems in production
- Advanced PySpark and Python engineering skills, with an explicit understanding of Spark architecture (lazy evaluation, DAG execution, execution memory vs. storage memory)
- High proficiency in advanced SQL for complex analytical troubleshooting
- Expert-level understanding of Delta Lake internals (ACID transactions, Time Travel, Liquid Clustering, Optimize/Z-Order commands)
- Proven track record of diagnosing and resolving distributed data pipeline issues at scale, including thread dumps, memory leaks, and serialization errors
- Strong architectural experience deploying and managing resources within major cloud networks (AWS, Azure, or GCP)
- Robust understanding of Git-based workflows, automated testing, and secure hotfix deployment via CI/CD pipelines
- Active Databricks Certifications (e.g., Databricks Certified Data Engineer Professional)
- Deep experience managing production environments utilizing Delta Live Tables (DLT) and advanced Unity Catalog rule engines
Benefits
- Medical, dental, and vision insurance with an employer contribution
- Flexible spending or health savings account
- Life and AD&D insurance
- Short and long term disability coverage
- Paid time off
- Employee assistance
- Participation in a 401k program with company match
- Additional voluntary or legally required benefits
- Remote role
Company Overview
Company H1B Sponsorship