[Remote] Big Data Engineer
Note: The job is a remote job and is open to candidates in USA. Software Guidance & Assistance, Inc. (SGA) is a technology and resource solutions provider seeking a Big Data Engineer for a contract assignment with a regulatory client. The role is responsible for designing, developing, and optimizing large-scale data processing systems, data pipelines, and integration solutions while ensuring platform performance, scalability, and reliability. The engineer will also collaborate with cross-functional teams, support data-driven decision-making, and monitor and troubleshoot production pipelines.
Responsibilities
- Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala)
- Implement data ingestion, storage, transformation, and analysis of solutions that are scalable, efficient, and reliable
- Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture
- Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions
- Optimize and enhance existing data pipelines for performance, scalability, and reliability
- Develop automated testing frameworks and implement continuous testing for data quality assurance
- Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines
- Work with data scientists and analysts to support data-driven decision-making across the organization
- Ability to write and maintain automated unit, integration, and end-to-end tests
- Monitor and troubleshoot data pipelines in production environments to identify and resolve issues
Skills
- Bachelor's degree in Computer Science, Information Systems or related discipline with at least five (5) years of related experience, or equivalent training and/or work experience
- Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions
- Past experience with developing enterprise quality solutions in an iterative or Agile environment
- Extensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworks
- Strong written and verbal technical communication skills
- Demonstrated ability to develop effective working relationships that improved the quality of work products
- Should be well organized, thorough, and able to handle competing priorities
- Ability to maintain focus and develop proficiency in new skills rapidly
- Ability to work in a fast paced environment
- Experience with object oriented programming languages such as Java, Scala or Python
- Hands-on experience with AI development tools (GitHub Copilot, Q Developer, ChatGPT, Claude, etc.)
- Strong software development background with ability to contribute to technical discussions
- Extensive experience with Scrum, Kanban, and continuous improvement practices
- Experience with Big data technologies such as Hadoop, Spark, Hive & Trino
- Data skew and strategies to mitigate it
- Working with massive data volumes in PetaBytes
- Troublehsooting job failures due to resource limitations, bad data, scalability challenged
- Proficiency in crafting effective prompts for AI coding assistants and analysis tools
- Experience redesigning development processes to leverage AI capabilities
- Ability to interpret AI-generated insights and translate them into actionable team improvements
- Experience leading teams through AI adoption and workflow transformation
- Assess comfort with SQL window functions, multi-table joins, aggregations
- Provide examples or ask them to write/optimize SQL queries on the spot
- Probe how they handle edge cases like NULLs, duplicates, ordering, etc
- Test their understanding of Spark's core architecture — executors, tasks, stages, DAG
- Focus on Spark performance tuning techniques: partitioning, caching, broadcast joins, etc
- Ask scenario-based questions on troubleshooting slow running/stuck jobs or resource issues in Spark
- Explore their experience optimizing Spark jobs for large-scale datasets
- Check exposure to AWS services like S3, EMR, Glue, Lambda, Athena, etc
- Ask how they've used S3 with Spark (e.g., dealing with file formats, consistency issues)
- EKS, Serverless knowledge, etc
- Assess ability to write clean, modular, and performant code
- Look for experience in functional programming concepts (e.g., immutability, higher-order functions)
- Ask about real-world use cases where they wrote scalable data processing code
- Evaluate understanding of collections, concurrency, and memory management
- Master's degree and past Financial Services industry experience preferred
Company Overview
Company H1B Sponsorship