Databricks Architect

Job Title: Databricks Architect Location: Remote Role Summary We are seeking an expert Principal Databricks Architect to lead the vision, design, and implementation of our next-generation enterprise Data Intelligence Platform. In this role, you will be the senior technical authority and trusted advisor responsible for building highly secure, scalable, and cost-effective data infrastructure across multi-cloud environments (Azure, AWS, and Google Cloud Platform). You will design robust Lakehouse architectures, implement enterprise-grade data governance, automate end-to-end ETL/ELT pipelines, and architect cutting-edge GenAI and machine learning infrastructure. This position requires a deep technical understanding of distributed systems, advanced cybersecurity controls, HIPAA compliance, and state-of-the-art MLOps/DataOps workflows to align technical strategy with business outcomes. Key Responsibilities 1. Data Platform Architecture & Engineering Lakehouse & Medallion Design: Architect and scale an enterprise-wide Databricks Lakehouse leveraging the Medallion Architecture (Bronze, Silver, Gold) to serve operational, analytics, and AI workloads. Pipeline Automation: Design and optimize real-time and batch data pipelines using Delta Live Tables (DLT), Apache Spark, PySpark, and Spark SQL. Storage Optimization: Enforce Delta Lake best practices, maximizing performance via transaction logs, ACID compliance, data compaction (OPTIMIZE), multi-dimensional clustering (Z-ORDER), and Auto Loader. Application Ecosystem: Oversee the development and integration of Databricks Apps to deliver data-driven tools directly within the platform ecosystem. 2. Enterprise Governance, Security & Compliance Unity Catalog Enforcement: Standardize data governance across the enterprise by implementing Unity Catalog for centralized access management, data lineage tracking, and auditing. Granular Security Controls: Design and automate advanced security rules, including Row-Level and Column-Level Security as well as dynamic data masking to safeguard sensitive assets. Compliance & RBAC: Build Automated Role-Based Access Control (RBAC) matrices integrated with cloud identity management. Ensure the entire platform architecture strictly complies with HIPAA compliance regulations for protective health information. Observability & Quality Gates: Implement automated Data Quality Gates to stop bad data from propagating, while monitoring operational health with advanced observability and auditing dashboards. 3. AI, Generative AI & MLOps RAG & GenAI Integration: Architect semantic search and generative AI pipelines using Databricks Vector Search, Azure OpenAI, Retrieval-Augmented Generation (RAG) frameworks, and customized Vector Embeddings. ML Lifecycle Management: Standardize machine learning workflows across the organization by deploying MLflow for model tracking, versioning, and environment reproducibility. MLOps Pipelines: Collaborate with data scientists to establish production-grade MLOps practices, automating model deployment, monitoring, and retraining pipelines. 4. Infrastructure, DevOps & Cloud Integration Multi-Cloud Ecosystems: Integrate Databricks seamlessly with cloud provider stacks, including Azure (Azure Databricks, ADLS Gen2, Synapse, Azure Data Factory), AWS, and Google Cloud Platform. Infrastructure as Code (IaC): Automate the orchestration, deployment, and configuration of Databricks workspaces and clusters using Terraform or other IaC tools. CI/CD Automation: Establish strict CI/CD and DataOps release engineering pipelines for notebooks, workflows, and infrastructure using Azure DevOps or GitHub Actions. Containerization: Utilize Docker to containerize application environments, testing frameworks, and custom execution environments. 5. FinOps & Platform Optimization Compute Tuning: Maximize system performance and reduce latency through targeted cluster optimization, custom cluster sizing, auto-scaling thresholds, Photon acceleration, and proactive instance selection. FinOps Governance: Define organizational cluster policies, budget alerts, and cost allocation models to eliminate wasted cloud expenditure. Required Skills & Qualifications Experience: 12+ years in Data Engineering/Architecture, with 3+ years of dedicated experience architecting enterprise-scale Databricks environments. Core Coding: Expert-level mastery of Python, PySpark, and complex SQL optimization (proficiency in Scala or Java is a plus). Data Security Expertise: Proven track record implementing Unity Catalog, Row/Column-level security, and building environments governed by HIPAA or similar compliance frameworks. Cloud Infrastructure: Strong background managing native storage and orchestrations across Azure (ADLS, ADF), AWS, or Google Cloud Platform. Modern Tools: Hands-on experience with Docker, Terraform, Databricks Workflows, and CI/CD pipeline automation. AI Frameworks: Practical architectural understanding of LLMs, Vector Databases, and MLflow. Certifications (Preferred): Databricks Certified Data Engineer or Platform Architect credentials