[Remote] Data Engineering Architect - Digital Operations

Note: The job is a remote job and is open to candidates in USA. Fractal is a strategic AI partner to Fortune 500 companies with a vision to power every human decision in the enterprise. The company is seeking an experienced Data Engineering Architect to provide operational excellence for enterprise-scale Azure Data and Analytics platforms, focusing on stability, reliability, performance, monitoring, incident management, root cause analysis, governance, and continuous improvement.

Responsibilities

• Provide L2/L3 support for Azure Data and Analytics platforms and services
• Manage and resolve production incidents, service requests, and critical issues within defined SLAs
• Monitor data pipelines, workflows, system health, and platform performance using Azure monitoring tools
• Analyze failures, data quality issues, and performance bottlenecks to ensure minimal business disruption
• Participate in on-call support and major incident management activities as required
• Perform detailed Root Cause Analysis (RCA) for recurring incidents and platform issues
• Implement corrective and preventive actions to improve platform stability
• Collaborate with engineering teams to deploy permanent fixes and operational enhancements
• Drive continuous improvement initiatives to reduce incident volumes and improve operational efficiency
• Ensure availability, scalability, and reliability of Azure-based data platforms
• Establish and enforce operational best practices, support procedures, and runbooks
• Maintain operational documentation, incident reports, and knowledge repositories
• Triage and resolve failed or stuck ADF pipeline runs, including debugging activity-level errors, dependency failures, and trigger misconfiguration
• Analyze integration runtime (IR) performance and connectivity issues between on-premises and cloud environments
• Optimize pipeline execution by reviewing parallelism settings, retry policies, and data movement activities
• Maintain and update linked services, datasets, and parameterized pipeline templates to support evolving operational needs
• Monitor and troubleshoot Databricks job clusters, workflows, and notebook executions in production environments
• Diagnose and resolve Spark job failures, including out-of-memory errors, executor loss, shuffle partitioning, and skewed data joins
• Optimize Databricks notebooks and jobs for performance, including Delta Lake read/write optimization, Z-ordering, VACUUM operations, and partition tuning
• Manage Databricks cluster configurations (autoscaling, spot instances, cluster policies) to balance cost and performance
• Automate recurring operational tasks using Databricks CLI, REST APIs, and Databricks Workflows (Jobs API)
• Investigate and resolve data ingestion failures, file format inconsistencies (Parquet, Delta, CSV), and storage permission errors
• Monitor ADLS Gen2 storage metrics — capacity usage, transaction rates, latency — and raise alerts for anomalies
• Manage hierarchical namespace configurations, access control lists (ACLs), and Azure RBAC assignments
• Coordinate data lifecycle management policies to control storage costs and data retention
• Build and maintain Azure Monitor dashboards, alerts, and Log Analytics (KQL) queries for proactive platform health monitoring
• Configure diagnostic settings and integrate platform logs with centralized monitoring solutions (Azure Monitor, Application Insights, or equivalent)
• Define and track SLA/SLO metrics for critical data pipelines and platform services
• Develop automated alerting for pipeline failures, SLA breaches, data quality anomalies, and infrastructure health events
• Implement and monitor data quality checks across ingestion, transformation, and serving layers
• Investigate data reconciliation issues, duplicate records, schema drifts, and late-arriving data scenarios
• Collaborate with engineering teams to embed automated data quality frameworks in production pipelines

Skills

• • 7+ years of experience in data engineering or analytics engineering roles, with at least 3 years in a production support or platform operations capacity
• • Strong communication and stakeholder management skills; ability to translate technical issues into clear business impact and provide timely incident updates to non-technical audiences
• • Demonstrated experience working within CPG, Retail, or a comparable domain with complex, high-volume data pipelines and tight SLA environments
• • Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related field (or equivalent practical experience)
• • Experience with CI/CD pipelines for data platform deployments using Azure DevOps or GitHub Actions, including branching strategies, release gates, and environment-specific configurations
• • Familiarity with Delta Live Tables (DLT) troubleshooting pipeline graph failures, data quality expectations, and incremental processing
• • Hands-on experience with Unity Catalog governance tasks, including access control, data lineage review, and schema management
• • Experience with infrastructure-as-code (IaC) practices using Terraform or ARM/Bicep templates for data platform provisioning
• • Advanced KQL skills for complex diagnostic queries and custom alerting rules beyond standard dashboards

Benefits

• Discretionary bonus for the current performance period (eligibility-based)
• Health insurance for full-time employees or hourly employees working more than 30 hours per week, in accordance with the plan documents
• Dental insurance for full-time employees or hourly employees working more than 30 hours per week, in accordance with the plan documents
• Vision insurance for full-time employees or hourly employees working more than 30 hours per week, in accordance with the plan documents
• Life insurance for full-time employees or hourly employees working more than 30 hours per week, in accordance with the plan documents
• Disability plans for full-time employees or hourly employees working more than 30 hours per week, in accordance with the plan documents
• Benefits eligibility on the first day of employment with the Company
• Eligibility to participate in the Company 401(k) Plan after 30 days of employment, in accordance with the applicable plan terms
• 11 paid holidays
• 12 weeks of Parental Leave
• “Free time” PTO policy, allowing flexibility to take the time needed for either sick time or vacation

Company Overview