[Remote] Data Engineer II

Note: The job is a remote job and is open to candidates in USA. NPS Prism is a cloud-based customer experience benchmarking and operational improvement platform owned by Bain & Company. The company is seeking a Data Engineer II to design, build, and optimize scalable data pipelines and cloud-based data solutions using Python, SQL, PySpark, Databricks, Azure, and AWS. The role also involves data quality, DevOps and CI/CD practices, stakeholder collaboration, documentation, and system optimization.

Responsibilities

• Design, build, and optimize ETL/ELT workflows using tools like Databricks, SQL, Python/pyspark & Alteryx (Good to have)
• Develop and maintain robust, scalable, and efficient data pipelines for processing large datasets. from source to emerging data
• Work on cloud platforms (Azure, AWS) to build and manage data lakes, data warehouses, and scalable data architectures
• Utilize cloud services like Azure Data Factory, AWS Glue, or for data processing and orchestration
• Use Databricks for big data processing, analytics, and real-time data processing
• Leverage Apache Spark for distributed computing and handling complex data transformations
• Create and manage SQL-based data solutions, ensuring high availability, scalability, and performance
• Develop and enforce data quality checks and validation mechanisms
• Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to deliver impactful data solutions
• Understand business requirements and translate them into technical solutions
• Leverage CI/CD pipelines to streamline development, testing, and deployment of data engineering workflows
• Work with DevOps tools like Git, Jenkins, or Azure DevOps for version control and automation
• Maintain clear documentation for data workflows, pipelines, and processes
• Optimize data systems for performance, scalability, and cost-efficiency

Skills

• * Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field
• * 3–6 years of experience in Data Engineering or related roles
• * Hands-on experience with big data processing frameworks, data lakes, and cloud-native services
• * Proficiency in Python, SQL, and PySpark for data processing and manipulation
• * ​​​​​​​Proven experience in Databricks and Apache Spark
• * ​​​​​​​Expertise in working with cloud platforms like Azure, AWS
• * Sound knowledge of ETL processes and tools like Alteryx. (Good to have)
• * Leveraging data lakes, data warehouses, and data pipelines
• * Data Pipeline – Build a Data Pipeline from scratch
• * Strong understanding of distributed systems and big data technologies
• * Basic understanding of DevOps principles and familiarity with CI/CD pipelines
• * Hands-on experience with tools like Git, Jenkins, or Azure DevOps
• * Familiarity with data visualization tools like Power BI, Tableau, or similar is a plus
• * Knowledge of streaming technologies such as Kafka or Event Hubs is desirable
• * Strong problem-solving skills and a knack for optimizing data solutions
• * Excellent communication (oral and written) skills

Benefits

• Work From Home Role

Company Overview

Company H1B Sponsorship