Data Engineer

Remote Full-time Now Hiring

2 Candidate Submittal Slots, New High Level Policy - MSP Owner: Rob Finton - Location: Washington, DC - Remote position but candidate may have to travel onsite to DC occasionally based on need - Duration: 6 months - Skill id: 10787969 - Need associates with at least 10 years of work exp with a minimum of 5 years of exp as data engineer - Skills: 8-10+ years experience required - MySQL - Databricks - Pyspark - Job description: - Design and develop scalable ETL/ELT pipelines using Databricks and PySpark - Build batch and real-time streaming ingestion frameworks - Develop reusable ingestion and transformation frameworks - Implement Medallion architecture (Bronze, Silver, Gold layers) - Develop incremental and CDC-based ingestion pipelines - Design and implement real-time streaming pipelines using Kafka and Structured Streaming - Optimize Spark jobs, SQL queries, and streaming pipelines - Implement Delta Lake-based ingestion and transformation frameworks - Tune partitioning, caching, and Spark execution strategies - Strong SQL and data modeling skills - Experience with cloud platforms and distributed systems - Familiarity with CI/CD pipelines and DevOps practices - Project Code:

Apply for This Position