[Remote] Databricks Data Engineer
Note: The job is a remote job and is open to candidates in USA. CloudTech Innovations is seeking an experienced Data Engineer/RSA to serve as a technical bridge between data platforms and enterprise clients. The role leads the architecture, implementation, optimization, and governance of Databricks Lakehouse platforms while advising clients, mentoring engineering teams, and managing multiple concurrent engagements.
Responsibilities
- Lead technical priorities across multiple enterprise client engagements simultaneously, balancing rapid delivery with scalable architecture and long-term platform strategy
- Architect and maintain automated CI/CD pipelines (GitHub Actions, Azure DevOps, Jenkins) for data engineering workflows, ensuring robust version control, automated testing, and seamless deployment of Databricks assets
- Serve as the primary technical advisor within client delivery teams, translating business challenges into scalable, production-ready data solutions while mentoring engineering teams and driving architectural decisions
- Architect, build, and optimize enterprise-scale Databricks Lakehouse platforms using Medallion Architecture (Bronze/Silver/Gold), supporting multiple business domains and large-scale production workloads
- Own the full development lifecycle, from technical design and ETL/ELT pipeline construction (Databricks Workflows, SQL Warehouses, Spark) to deployment, monitoring, and production support
- Implement governed Lakehouse patterns using Unity Catalog, RBAC/ABAC, lineage, and compliance controls while balancing security with development agility
- Design distributed integration patterns across cloud-native services (AWS / Azure / GCP) and enterprise systems (CRM, ERP, Kafka/Kinesis streams)
- Lead architecture reviews, mentor engineering teams, establish platform best practices, and drive cloud cost optimization across storage, compute, and enterprise data platforms
Skills
- 10–12+ years of experience in Data Engineering, Cloud Engineering, or Solution Architecture
- 5+ years of hands-on experience with Databricks Lakehouse Platform
- Experience leading enterprise-scale cloud data platform implementations
- Strong client-facing experience working directly with technical and business stakeholders
- Experience mentoring engineering teams and leading technical design discussions
- Strong hands-on experience with Unity Catalog, Delta Live Tables (DLT), Photon Engine, and Lakehouse Federation
- Demonstrated experience implementing CI/CD pipelines for data platforms
- Proficiency with Terraform and version control workflows
- Proficiency in Spark (PySpark & Scala), complex query tuning, performance optimization, and pipeline automation (Airflow, dbt)
- Hands-on experience with cloud infrastructure (AWS / Azure / GCP), containerization (Docker), and building/debugging data-connected applications (FastAPI, Flask, Node.js)
- Experience with real-time streaming technologies (Kafka, Kinesis, Pub/Sub, Auto Loader)
- Certifications in Databricks, AWS/GCP/Azure Solution Architecture, or TOGAF are a plus
Benefits
- Remote work arrangement
Company Overview