[Remote] Senior Data Engineer – Databricks
Note: The job is a remote job and is open to candidates in USA. HireTalent is recruiting for a confidential company seeking a Senior Data Engineer to build and maintain scalable, reliable data infrastructure. The role focuses on Databricks-based data pipelines, lakehouse architecture, data quality, governance, integrations, and reusable data products supporting analytics, investment workflows, and AI applications.
Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL
- Build and optimize data processing workflows using Apache Spark and Delta Lake
- Develop reliable data ingestion, transformation, validation, and enrichment processes across multiple data sources
- Work with large and complex datasets and optimize pipelines for performance, scalability, and cost efficiency
- Build reusable data products and data services that support analytics, investment workflows, and AI applications
- Implement data quality, validation, reconciliation, and monitoring processes to ensure data accuracy and reliability
- Contribute to data architecture and lakehouse design decisions
- Work with data governance, metadata, lineage, and access-control requirements
- Collaborate with Data Architects, Product Managers, Software Engineers, Data Scientists, and business stakeholders to translate requirements into scalable data solutions
- Integrate data from APIs, databases, files, and third-party data providers
- Develop and maintain automated testing and data-quality frameworks
- Support CI/CD practices and automated deployment of data engineering workloads
- Troubleshoot complex data and pipeline issues and continuously improve platform reliability
- Contribute to the evolution of Rimes’ AI-ready data infrastructure and intelligence fabric
- Build highly scalable and reliable data pipelines
- Improve the quality, accessibility, and usability of investment data
- Create governed and reusable data products
- Improve data-processing performance and operational efficiency
- Strengthen the foundation required for AI-powered investment workflows
- Help transform complex capital-markets data into trusted, actionable intelligence
Skills
- 6+ years of experience in data engineering or a related discipline
- Strong hands-on experience with Databricks
- Strong programming experience with PySpark and Python
- Advanced SQL skills, including query optimization and performance tuning
- Strong experience building production-grade data pipelines and ETL/ELT workflows
- Experience with Delta Lake / Delta tables and modern lakehouse architectures
- Strong understanding of data modeling, data integration, and distributed data processing
- Experience working with large-scale datasets in a cloud environment
- Experience with data quality, validation, monitoring, and troubleshooting
- Strong understanding of software engineering practices, including version control, testing, and CI/CD
- Excellent problem-solving and communication skills
- Experience with Databricks Unity Catalog and data governance
- Experience with Delta Sharing or data-sharing technologies
- Experience with Databricks Workflows, Jobs, or similar orchestration capabilities
- Experience with AWS or Azure cloud platforms
- Experience with technologies such as AWS Glue, S3, Azure Data Factory, ADLS, or similar cloud data services
- Experience with Snowflake or other modern cloud data platforms
- Experience working with REST APIs and third-party data providers
- Experience with Terraform or infrastructure-as-code
- Experience implementing data lineage, metadata management, and data governance
- Financial services, investment management, capital markets, or market-data experience
- Exposure to AI/ML data pipelines or GenAI applications
Company Overview