Databricks & AI Engineer
PTN_US_GBAMSREQID_CandidateBeelineID MSP Owner: Michelle Lee Location: Remote Duration: 6 months skill id: 10892152 Role Overview We are seeking a highly skilled Databricks & AI Engineer to join our growing FinOps team. In this role, your primary focus will be engineering-driven: building robust data ingestion pipelines in Databricks, managing massive datasets across multi-cloud environments using Lakehouse architecture, and leading our transition toward Agentic AI Proof of Concepts (PoCs). While this role sits within the FinOps functional area, core FinOps domain knowledge is secondary. We are looking for a pure technical powerhouse who can leverage Python, Databricks, and advanced AI frameworks to optimize, track, and revolutionize how we manage cloud spend.
Key Responsibilities
Data Pipeline Engineering - Design, build, and maintain scalable, automated data ingestion pipelines to collect cloud billing, usage, and performance data from various sources in Databricks. Big Data Processing - Leverage Databricks (Spark/PySpark) to process, transform, and clean large-scale unstructured and structured cloud financial data. API Integration - Develop and maintain API integrations (specifically utilizing FastAPI) to pull data from diverse vendor platforms and internal systems. Multi-Cloud Data Management - Query and analyze massive datasets across AWS (Athena), GCP (BigQuery), and Azure (Data Lake & Cosmos DB). AI & Automation Innovations - Architect and implement next-generation Agentic AI workflows and PoCs to automate cloud cost anomalies, forecasting, and recommendations. Version Control & DevOps - Manage codebase integrity and CI/CD alignment using Git and its associated ecosystem. ________________________________________________ Required Skills Data & Systems Engineering - Hands-on experience building production-grade data pipelines. - Experience working natively with large datasets using distributed computing (Spark/PySpark). Big Data Ecosystem - Strong working knowledge of Databricks. - Familiarity with Data Lake architecture. - Familiarity with Medallion Architecture. Programming & Querying - Advanced proficiency in Python. - Advanced proficiency in SQL. API Development - Proven experience creating and consuming RESTful API calls to aggregate data. Cloud Architecture - Solid understanding of multi-cloud platforms, specifically: - Azure Data Lake - Cosmos DB - AWS - GCP Version Control - Proficient with Git. - Standard branching and collaboration workflows. ________________________________________________
Preferred Qualifications
Agentic AI - Hands-on experience or deep conceptual understanding of Agentic AI frameworks, specifically: - LangChain - LangGraph Web Frameworks - Experience with FastAPI. Cloud Data Warehousing - Experience with: - GCP BigQuery - AWS Athena FinOps Familiarity - Basic understanding of cloud economics. - Billing structures. - FinOps framework principles. Skills: - Digital: Databricks ~ Data Build Tool Experience Required: 8-10