AI Operations Specialist
About the Role
We are seeking an AI Operations Specialist to help business and product teams use AI to take on review and quality work. You will map how work gets done today, build AI-driven workflows, and test them until the results can be trusted. You will also help product teams decide whether an AI feature is ready to launch. This is a hands-on role: you help build the workflow, ship it, and stay on to make sure it keeps working. The scope is expected to grow as AI capabilities evolve.
Key Responsibilities
Help prioritize projects by business impact and make the case for the ones worth pursuing. Partner with process owners to map current workflows, then define where AI can do the work and where people step in on hard cases. Build AI workflows: write prompts, set up pipelines, maintain agreed quality levels, and monitor for drift after launch. Build test sets, measure accuracy, and calibrate LLM judges against human reviewers. Help product teams define launch-readiness criteria for AI features and provide evidence for go/no-go decisions. Turn unclear policy into criteria reviewers can apply consistently, working with product, legal, and policy partners. Work with subject matter experts to write labeling guidelines and produce human-verified labels on real data. Run labeling at scale with vendor agents, and validate their work using agreement metrics and spot checks. Make solutions reusable and document learnings so future teams reach production faster. Keep methods current as models, tools, and vendor agents change, and retire what no longer works. Own assigned workstreams end to end, from setup through quality checks.
Qualifications
Required 5-7 years in operations, program management, analytics, consulting, or applied AI, with accountability for results. Daily AI user who has built something with AI that delivered measurable impact, such as a workflow, an automation, or an analysis. Comfortable with metrics such as accuracy, precision, recall, and sampling. Experience taking a messy process and making it measurable. Quality experience in some form, such as review guidelines, QA, or audits. Clear writer, able to work with business leads, product managers, and engineers. Self-directed and comfortable working with limited direction. A technical background is not required. Prior experience designing evaluations or calibrating LLM judges is not required; training is provided. Preferred Hands-on SQL, Python, or API experience. Trust and safety, content moderation, annotation operations, or other human-in-the-loop work at scale. Experience fine-tuning models or calibrating LLM judges. Background in a two-sided marketplace, travel, or another operations-heavy business. Experience setting up intake and prioritization for a new team. Key Details Start date: November 23, 2026 Duration: 12 months Location: Remote (within the U.S.) Work schedule: 40 hours/week (8 hours/day, 5 days/week) Pay rate range: $65-$75/hour + benefits Work Authorization: Must be authorized to work in the U.S. without sponsorship. Engagement: W-2 Contract