[Remote] AWS Fullstack Data Engineer
Note: The job is a remote job and is open to candidates in USA. Smart IT Frame LLC is seeking an experienced Senior Full-Stack Data Engineer to lead the architecture, development, and optimization of a next-generation data platform. The role focuses on building scalable data pipelines, managing cloud-native data infrastructure, optimizing compute engines, and implementing security, observability, and real-time data solutions while providing technical leadership and mentorship.
Responsibilities
- Data Pipeline Development & Management
- Ingestion & Transformation: Design, build, and optimize high-volume data ingestion and transformation jobs using tools like dbt Core, AWS Glue, ensuring data quality and integrity
- Workflow Orchestration: Develop and maintain sophisticated data pipelines using orchestrators such as Dagster, focusing on modularity and reusability
- Streaming & Real-time Integration: Implement and manage real-time data flows utilizing Confluent platforms or native AWS streaming services (e.g., Kinesis) for immediate data availability
- Data Security and Privacy: Data Anonymization, Compliance with Regulations
- AWS Architect development for Data Pipeline
- Be well versed with DataOps and DevOps fundamentals
- Assist and drive the Data Ecosystem Management & Monitoring
- Has experience with containerization and orchestration, specifically AWS ECS/ EKS
- Infrastructure as Code (IaC): Can help with development and maintenance of some cloud foundation using IaC tools to ensure immutable, repeatable, and scalable deployments
- Open Table Formats & Management: Implement and maintain the Iceberg open table format, utilizing tools for efficient schema evolution and data management
- Compute Engine Optimization: Optimize query performance and cost efficiency across our primary compute engines: Snowflake, Amazon Redshift, and AWS Athena
- Observability & Monitoring: Integrate comprehensive monitoring and observability into all pipelines using Splunk to ensure high availability, rapidly identify bottlenecks, and troubleshoot production issues
Skills
- • 15+ Years of hands-on, progressive experience in Data Engineering, Data Architecture, or a closely related Full-Stack Data role
- • Deep conceptual understanding of core data engineering principles, ETL/ELT patterns, and metadata management
- • Proven track record of building and managing petabyte-scale data infrastructure in a cloud-native environment
- • Insurance industry experience is mandatory
- • Cloud Environment: AWS (S3, IAM, VPC, etc.)
- • Experience with Talend, dbt Core, Iceberg, AWS Glue Catalog, Snowflake, Redshift, Athena, Splunk, AWS streaming services, Git
- • Strong SQL, Pyspark and Python
Company Overview
Company H1B Sponsorship