Senior Data Annotator (Portland, Remote)
Job Opening: Senior Data Annotator (Portland, Remote) Rex.zone is hiring a Senior Data Annotator to produce high-quality labeled data and evaluation signals used to train and assess AI/ML models. You will support LLM training pipelines, RLHF preference data, prompt evaluation, QA evaluation audits, and content safety labeling while documenting edge cases to improve training data quality and model performance.
What You Will Do
- Perform expert-level data labeling across modalities: NLP text classification, named entity recognition, summarization checks, and dialogue evaluation for LLMs
- Create RLHF preference judgments and rationales for pairwise ranking, instruction following, and helpfulness/harmlessness criteria
- Run QA evaluation workflows: spot checks, inter-annotator agreement reviews, adjudication support, and error taxonomies
- Annotate computer vision data when needed (bounding boxes, polygons, keypoints) and verify label accuracy against guidelines
- Conduct prompt evaluation and model output grading for factuality, safety, tone, policy compliance, and refusal quality
- Document edge cases, propose guideline updates, and provide clear feedback to improve dataset consistency
Required Qualifications
- Mid-Senior experience in data annotation, data labeling, or QA evaluation in production environments
- Ability to interpret and apply complex annotation guidelines with high consistency
- Hands-on experience with LLM evaluation, RLHF, prompt evaluation, or human feedback workflows
- Strong written reasoning for explaining labels, preferences, and safety decisions
- Comfort working remotely with versioned instructions, batch tracking, and deadlines
Employment Details
- Workplace: Remote (Portland-based candidates welcome; work is performed remotely)
- Employment Type: Full-time
- Compensation: $30–$50 per hour