Freelance Agent Evaluation Engineer

About the role

Mindrift is building a dataset to evaluate AI coding agents on realistic developer tasks. As a Freelance Agent Evaluation Engineer you will design, implement, and validate challenging tasks that test the limits of current AI models.

Key responsibilities

• Create realistic developer environments (codebase, infrastructure, tickets, docs, conversations) that simulate a believable development history.
• Design tasks from intermediate states, write clear prompts, define success criteria, and ensure they are solvable by an AI agent.
• Write functional and integration tests that accept all correct solutions while rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback, reviewing agent solutions and refining evaluation criteria.

Required profile

• 5+ years of professional software development experience.
• Strong English proficiency (minimum B2).
• Ability to understand where AI models fail and craft scenarios that expose those weaknesses.

Required skills

• Python (FastAPI)
• JavaScript/TypeScript (React)
• Docker
• PostgreSQL
• Kafka
• Redis
• Writing functional and integration tests

What we offer

• Freelance contract with flexible schedule.
• Compensation up to $50 per hour, based on experience and pace.
• Opportunity to work on cutting‑edge AI evaluation projects for leading tech companies.