[Remote] Data Engineer

Note: The job is a remote job and is open to candidates in USA. Rubicon is a company with an Information Technology department supporting analytics, business, and AI initiatives. The Data Engineer will build and operate production data pipelines on Databricks, collaborating with cross-functional teams to deliver reliable, documented, and governed data solutions from requirements through production. The role also owns pipeline quality, monitoring, CI/CD, governance, and on-call operations.

Responsibilities

• Build batch and streaming ingestion pipelines from databases, APIs, SaaS applications, event streams, and file drops using Lakeflow Declarative Pipelines for CDC and SCD Type 2 patterns
• Build and maintain medallion architecture (bronze, silver, gold) with clear contracts at each layer, applied as part of every pipeline you deliver
• Write and tune complex SQL and PySpark transformations against large Delta Lake tables, and model gold-layer datasets for analytics
• Own operational health of your pipelines, including data quality checks, monitoring and alerting, Unity Catalog governance, Continuous Integration/Continuous Development (CI/CD) promotion, and on-call
• Ability to travel and/or work onsite as needed
• Performs other duties as assigned or apparent
• This job has no direct supervisory responsibilities
• May assist with onboarding and training new hires on Information Technology processes, systems, and operational procedures
• Serves as a resource for internal team members regarding IT/Data operational processes and standards
• Supports knowledge sharing, continuous improvement initiatives, and best practices across the Information Technology and Data Engineering functions

Skills

• High School Diploma required
• 5+ years of experience in Data Engineering or adjacent roles with a track record of owning and fixing pipelines in production
• Strong knowledge of Production Apache Spark (PySpark) and Python (modular, testable code with unit tests)
• Demonstrated understanding of data modeling and pipeline mechanics, including dimensional modeling, grain and keys, slowly changing dimensions, CDC, schema drift, and formats such as Parquet and Avro
• Proficiency with Delivery fundamentals (orchestration with Lakeflow Jobs, Airflow, or Dagster; Git and code review; Continuous Integration/Continuous Deployment (CI/CD); cloud object storage on a major cloud provider)
• Advanced SQL skills, including window functions, CTEs, complex joins, aggregations over large datasets, and identifying why queries are slow through query plan analysis
• An in-depth understanding of incremental processing, idempotency, and late-arriving data
• Experience with debugging correctness issues by reasoning through joins, grain, and duplication
• Hands on Databricks experience required, including:
• Delta Lake at scale: MERGE-based upserts, OPTIMIZE, time travel, schema evolution
• Lakeflow Declarative Pipelines and Lakeflow Jobs. Prior Delta Live Tables and Workflows experience counts
• Unity Catalog for governance, access control, and lineage
• Databricks SQL warehouses and compute selection, including cost awareness
• Ability to communicate with stakeholders at all levels, ensuring action can be taken from written reports without follow-up meetings
• Works well with analysts, product owners, and source system teams to turn ambiguous requirements into workable specifications
• Scopes work into shippable increments
• Strong organizational and project management skills with the ability to manage multiple priorities and deadlines
• A proactive, can-do attitude with a willingness to take ownership of tasks and drive them to completion
• Ability to work independently while also being a team player who thrives in a collaborative environment
• Discretion and trustworthiness in handling sensitive information and supporting high-level strategic initiatives
• Ability to travel and/or work onsite as needed
• While performing the duties of this job in a home office setting, the employee is regularly required to work on a computer for extended periods of time
• Frequent use of a computer requires fine motor skills and hand-eye coordination
• Ability to sit for extended periods while working from home or a designated workspace
• Ability to perform tasks that require sustained attention and focus
• Occasional lifting of materials up to 25 pounds
• Travel to attend team meetings may be required
• To facilitate working from home, and as a requirement for this role, the employee must provide reliable internet connection with sufficient bandwidth to execute all job functions and technology setup conducive to remote work
• A quiet, distraction-free workspace is required for maintaining productivity
• Bachelor's degree in Business, Information Systems, Data Science, or related field preferred
• Databricks Certified Data Engineer Associate or Professional certifications are highly valued
• Lakeflow Connect or another managed ingestion or CDC tool, plus Databricks Asset Bundles, Terraform, or Docker
• Data quality and observability tooling, BI tools and reverse ETL, and PII masking and retention practices
• RAG, vector search, or document processing workloads, and MLflow tracing and evaluation
• No prior AI experience is required, but GenAI Engineering experience (document ingestion pipelines, vector search indexes, and MLflow tracing and evaluation) is valued

Benefits

• Remote/hybrid work arrangement in the USA
• The company laptop will be provided

Company Overview

Company H1B Sponsorship