Databricks SME (Subject Matter Expert)
This is a remote position.
KATBOTZ is seeking an experienced Databricks SME to support enterprise-level data engineering and analytics initiatives. The ideal candidate will have deep expertise in the Databricks ecosystem, big data technologies, cloud platforms, and modern data architecture. The candidate will work closely with cross-functional teams to design, optimize, and implement scalable data solutions for global clients.
Key Responsibilities
Lead the design, development, and optimization of Databricks-based data platforms and pipelines.
Provide technical expertise on Databricks architecture, performance tuning, and best practices.
Build scalable ETL/ELT workflows using Apache Spark and Databricks.
Collaborate with business stakeholders, architects, and engineering teams to gather requirements and deliver solutions.
Implement Delta Lake, data governance, and data quality frameworks.
Support migration of legacy data platforms to Databricks.
Optimize data processing workflows for performance and cost efficiency.
Mentor development teams on Databricks, Spark, and cloud-native solutions.
Troubleshoot production issues and provide timely resolutions.
Ensure adherence to security, compliance, and coding standards.
Lead the design, development, and optimization of Databricks-based data platforms and pipelines.
Provide technical expertise on Databricks architecture, performance tuning, and best practices.
Build scalable ETL/ELT workflows using Apache Spark and Databricks.
Collaborate with business stakeholders, architects, and engineering teams to gather requirements and deliver solutions.
Implement Delta Lake, data governance, and data quality frameworks.
Support migration of legacy data platforms to Databricks.
Optimize data processing workflows for performance and cost efficiency.
Mentor development teams on Databricks, Spark, and cloud-native solutions.
Troubleshoot production issues and provide timely resolutions.
Ensure adherence to security, compliance, and coding standards.
Required Skills & Qualifications
10+ years of experience in Data Engineering and Big Data technologies.
Strong hands-on expertise in Databricks and Apache Spark.
Experience with Python, PySpark, SQL, and Scala.
Expertise in Delta Lake, Unity Catalog, and Databricks Workflows.
Strong understanding of data warehousing and modern lakehouse architecture.
Experience with cloud platforms such as Microsoft Azure, Amazon AWS, or Google GCP.
Experience integrating Databricks with BI and reporting tools.
Knowledge of CI/CD pipelines and DevOps practices.
Strong analytical and problem-solving skills.
Excellent communication and stakeholder management abilities.
Preferred Qualifications
Databricks Certified Professional certification.
Experience in real-time streaming using Kafka or Structured Streaming.
Exposure to AI/ML workflows within Databricks.
Prior experience working with global remote teams.
Benefits
- Competitive compensation package
- Opportunities for professional development and career advancement.
- Flexible working conditions, with remote options available.
- Dynamic and supportive work environment.
Equal Employment Opportunity
10+ years of experience in Data Engineering and Big Data technologies.
Strong hands-on expertise in Databricks and Apache Spark.
Experience with Python, PySpark, SQL, and Scala.
Expertise in Delta Lake, Unity Catalog, and Databricks Workflows.
Strong understanding of data warehousing and modern lakehouse architecture.
Experience with cloud platforms such as Microsoft Azure, Amazon AWS, or Google GCP.
Experience integrating Databricks with BI and reporting tools.
Knowledge of CI/CD pipelines and DevOps practices.
Strong analytical and problem-solving skills.
Excellent communication and stakeholder management abilities.
Databricks Certified Professional certification.
Experience in real-time streaming using Kafka or Structured Streaming.
Exposure to AI/ML workflows within Databricks.
Prior experience working with global remote teams.
KATBOTZ LLC is an Equal Opportunity Employer. We provide equal employment opportunities to all qualified individuals, regardless of race, religion, gender, gender identity, age, marital status, national origin, sexual orientation, citizenship status, veteran status, disability, or any other legally protected status. As an organization, we are unwavering in our commitment to maintaining a discrimination-free work environment, and fostering a culture of inclusivity, belonging and equal opportunity for all employees and applicants.
Originally posted on