[Remote] AI Engineering Manager Machine Learning
Note: The job is a remote job and is open to candidates in USA. TalentOla is seeking an AI Engineering Manager focused on machine learning and AI systems. The role involves designing, building, deploying, and owning AI-powered features and agent-based workflows, while driving experimentation, reliability, safety, and collaboration across engineering and product teams.
Responsibilities
- Design and implement AI-powered features (LLM workflows, copilots, and agent-based systems with tool use and multi-step reasoning)
- Own the Full Lifecycle: prototyping → evaluation → production deployment → iteration
- Ensure solutions are reliable, performant, and aligned with product needs
- Build and optimize prompt pipelines for specific use cases
- Build retrieval systems (embeddings, chunking, ranking)
- Implement RAG-based workflows where needed
- Iterate on outputs to improve quality, accuracy, and consistency
- Design scalable and cost-efficient AI architectures for production workloads
- Select and evaluate models (hosted vs open-source) based on use case constraints
- Design and build agentic workflows capable of multi-step reasoning and decision-making
- Integrate agents with tools, APIs, and internal systems to perform real-world actions
- Implement planning, execution, and reflection loops for complex tasks
- Manage context, memory, and state across multi-step interactions
- Balance deterministic workflows vs. agent autonomy for reliability and control
- Run structured experiments to compare approaches (prompting, retrieval, models)
- Define and track key metrics for AI performance (quality, latency, cost)
- Debug and improve non-deterministic system behavior
- Build and maintain evaluation datasets and benchmarks
- Implement automated evaluation pipelines for continuous improvement
- Drive technical direction and influence AI adoption across teams
- Partner with product managers and designers to scope AI features
- Contribute to shared patterns and reusable components
- Participate in code reviews and design discussions
- Support and mentor mid-level engineers where needed
- Design guardrails to ensure safe and reliable AI behavior
- Mitigate hallucinations, prompt injection, and model misuse
- Ensure compliance with data privacy and enterprise requirements
- Implement monitoring and observability for AI systems in production
- Implement guardrails for agent actions (tool access control, execution boundaries)
- Prevent failure cascades in multi-step agent
Skills
- Strong understanding of LLM capabilities and limitations
- Experience with prompt engineering and structured output design
- Hands-on experience with embeddings and vector search
- Familiarity with RAG architectures and when to apply them
- Experience designing agent-based architectures (AgentCore concepts)
- Understanding of tool use, planning strategies, and memory mechanisms in LLM systems
- 4+ years of related work experience
- Solid backend/system design fundamentals
- Experience building and deploying production-grade systems
- Ability to debug complex issues, including probabilistic outputs
- Comfort working with APIs, pipelines, and data flows
- Ability to translate user needs into effective AI solutions
- Strong intuition for balancing quality, latency, and cost
- Focus on delivering measurable product impact
- Communicates clearly across engineering and product teams
- Contributes to team knowledge and shared practices
Company Overview