[Remote] Technical Lead for Data engineering
Note: The job is a remote job and is open to candidates in USA. Confidential is seeking a skilled and motivated Data Engineer to support data infrastructure built on AWS services. The role is responsible for monitoring, troubleshooting, optimizing, and maintaining data pipelines, storage, data quality, and operational workflows while collaborating with engineering, analytics, and operations teams.
Responsibilities
- Monitor and troubleshoot data pipelines and workflows utilizing AWS Glue, Step Functions, and Lambda. Optimize existing data pipelines for performance and cost efficiency
- Manage and support data stored in Amazon S3 and ensure efficient storage policies (e.g., lifecycle rules, versioning, and encryption). Ensure optimal performance and availability of Amazon Redshift clusters, including schema maintenance and query tuning
- Respond to incidents related to data failures, latency, or data quality and provide timely resolutions. Debug and resolve issues in ETL jobs, data ingestion, and transformations
- Perform root cause analysis for recurring issues and implement solutions to enhance the reliability of the data ecosystem. Identify areas for process improvement and implement automation wherever feasible
- Implement monitoring tools and dashboards to track data pipeline health. Collaborate with stakeholders to ensure adherence to data security, governance, and compliance policies
- Maintain up-to-date documentation for data pipelines, workflows, and troubleshooting steps. Provide regular reports on system performance and key metrics
- Work closely with data engineering, analytics, and operations teams to resolve issues and gather requirements for enhancements. Support ad-hoc data requests and ensure timely delivery of data to business users
Skills
- Minimum 7 yrs in Data engineering with relevant skills
- AWS Services (S3, Redshift, Lambda, Glue, Step function) Python
- Proficient in AWS services: S3, Redshift, Glue, Step Functions, Lambda
- Strong understanding of ETL/ELT processes and data transformation
- Experience with monitoring and debugging data pipelines in a production environment
- Proficiency in SQL and hands-on experience with Redshift for data modeling and performance tuning
- Proficiency in Python for scripting, data manipulation, and building serverless applications with Lambda
- Any Graduation
- Knowledge of other databases (e.g., PostgreSQL, MySQL) is a plus
- Knowledge of PySpark or similar frameworks is a plus
Company Overview
Company H1B Sponsorship