[Remote] Software Engineer - AI Coding Agent Evaluation
Note: The job is a remote job and is open to candidates in USA. Mindrift connects specialists with project-based AI opportunities for leading technology companies, focused on testing, evaluating, and improving AI systems. The Software Engineer will design realistic coding tasks for AI coding agents, creating broken software environments, automated tests, instructions, and reference solutions, while calibrating task difficulty and incorporating QA feedback.
Responsibilities
- Invent a realistic developer scenario — a real bug, a broken ETL, a missing feature — not a toy problem
- Build a reproducible Docker environment with pinned dependencies
- Write a pytest that verifies outcomes, not specific commands — deterministic, non-flaky, and does not leak the fix
- Write an instruction.md that reads like a Jira ticket a developer would receive
- Write a reference solve.sh proving the task is solvable
- Calibrate difficulty so current state-of-the-art agents solve the task 20–60% of the time
- Iterate based on feedback from expert QA reviewers
- Later: review other authors’ tasks as a QA reviewer
Skills
- *Please submit your CV in English and indicate your level of English proficiency.*
- * 3+ years of production software development in one backend stack — Python, Go, Node.js, Java, or Rust. Depth in one stack beats breadth
- * Python + pytest fluency — required regardless of primary stack. The task harness is pytest-based even when the broken app is in another language. Fixtures, parametrize, monkeypatch, timeouts, conftest.py
- * Docker authoring — reproducible Dockerfiles, pinned dependencies, multi-stage builds when needed, non-root user
- * Linux & Bash — comfort debugging inside containers (strace, lsof, journalctl); shell beyond set -euo pipefail
- * AI coding agent experience — Claude Code, Cursor, Roo Code, or similar, on non-trivial work. You can cite a specific time the AI was confidently wrong and how you caught it
- * English — B2+ written
- * Domain depth in Security, System Administration (nginx / systemd / cron), Scientific Computing (NumPy / PyTorch / SciPy), DevOps, or Git internals
- * Modern Python tooling (uv, poetry, pyproject.toml)
- * Coverage tooling (pytest-cov, coverage.py, gcov, llvm-cov, kcov)
- * Fuzzing or property-based testing (Hypothesis)
- * Prior contribution to agent-evaluation benchmarks or related frameworks
Benefits
- Some projects include incentive payments.
- You choose when and how to contribute; tasks must be submitted by the deadline and meet acceptance criteria.
Company Overview