Remote Data Annotation Specialist in New York
Remote Data Annotation Jobs in New York Rex.zone is hiring for a full-time, remote Data Annotation Specialist role supporting AI/ML training pipelines. You will label and evaluate text, images, and conversational outputs used in LLM training, RLHF, prompt evaluation, and training data quality programs. About The Role You will perform remote data annotation and evaluation tasks that directly improve model performance. Projects may include NLP datasets (classification, named entity recognition, retrieval relevance), conversational AI (prompt evaluation and response grading), computer vision annotation (bounding boxes, polygons, segmentation), and content safety labeling.
What You Will Do
- Apply labeling taxonomies and annotation guidelines to text, image, and multimodal data
- Perform RLHF-style preference ranking and prompt evaluation for large language model evaluation
- Conduct QA evaluation, audits, and adjudication to resolve edge cases and improve inter-annotator agreement
- Document rationales and error patterns that drive model performance improvement
- Collaborate with project leads to refine rubrics, update policies, and calibrate labeling decisions
Requirements
- Professional experience with data annotation, data labeling, evaluation, or QA workflows
- Strong written communication and ability to justify decisions using rubrics
- Comfort with annotation tools, spreadsheets, and web-based labeling platforms
- Ability to maintain high accuracy under detailed policies and changing guidelines
- Reliable remote setup and ability to meet full-time productivity and quality targets
Quality & Success Metrics Success is measured by training data quality, annotation guidelines compliance, throughput vs. targets, and consistency on edge cases. Metrics may include accuracy vs. gold data, QA pass rates, inter-annotator agreement, rework percentage, and calibration performance. Location Eligibility This is a remote role typically performed from anywhere in New York State (and sometimes broader US eligibility depending on project requirements). Always follow the Rex.zone posting details for location eligibility.