DataBricks Architect
Title: DataBricks Architect Location: Remote (EST candidates only) Duration: Long Term Contract Experience Required: Min 8 years Architect total 15Y IT Need Healthcare Domain Experience Job Description: Responsible for designing, building, and optimizing enterprise-scale data platforms using the Databricks Lakehouse architecture, Delta Lake, Apache Spark, and related technologies. The role bridges business needs with technical solutions, ensuring scalable, secure, and costefficient data systems. The role combines technical depth in data engineering and cloud architecture with strategic thinking to deliver scalable, secure, and innovative data platforms. Design & Implement Data Platforms: Create scalable data architectures using Databricks, Delta Lake, and Spark, translating business requirements into productionready solutions. ETL/ELT Pipeline Development: Build and optimize batch and streaming pipelines for performance, reliability, and cost control, Cluster & Resource Management: Define cluster sizing, configuration, autoscaling, and costcontrol strategies; monitor and tune resource utilization. Data Modeling & Governance: Establish best practices for schema design, partitioning, data modeling, and Delta Lake transaction management; implement security, governance, and access controls (e.g., Unity Catalog, encryption, rolebased access). Healthcare Data Expertise: Leverage deep domain knowledge to architect solutions that handle sensitive healthcare datasets (e.g., EHR, claims, RWE), ensuring strict adherence to HIPAA, GxP, and data privacy regulations while maintaining audit trails for compliance. Strategic Technical Leadership: Act as the trusted technical authority for the client, leading design reviews, advising on roadmap evolution, and translating complex business outcomes into scalable Lakehouse blueprints that align with J&J s long-term data strategy. Performance & Cost Optimization: Drive continuous improvement by implementing advanced optimization techniques including Auto-Optimize, Z-Ordering, and Photon acceleration to reduce query latency and cloud spend without compromising SLAs. Collaboration & Mentorship: Partner closely with data scientists, engineers, and business stakeholders to define data product requirements, while mentoring junior team members on Databricks best practices, code quality, and deployment standards. CI/CD & Automation: Champion Infrastructure-as-Code (IaC) and automated deployment pipelines (e.g., Terraform, Azure DevOps) to ensure reproducible, version-controlled, and resilient environments across development, test, and production stages.