[Remote] AI Test Engineer
Note: The job is a remote job and is open to candidates in USA. Vaisesika is seeking an AI Testing Engineer to test and evaluate Generative AI, LLM, RAG, and AI Agent applications. The role focuses on building automated evaluation frameworks, validating model responses, monitoring LLM performance, and integrating AI testing into CI/CD pipelines.
Responsibilities
- Design and execute testing strategies for Generative AI, LLM, RAG, and AI Agent applications
- Perform LLM evaluation and quality testing using DeepEval
- Build and maintain evaluation frameworks, test datasets, and automated test cases
- Use Langfuse for LLM observability, tracing, prompt management, and evaluation
- Validate LLM responses for accuracy, relevance, faithfulness, hallucination, toxicity, and consistency
- Develop automated tests for AI/LLM applications using Python
- Evaluate RAG pipelines for retrieval accuracy and response quality
- Perform prompt testing, regression testing, functional testing, and performance testing of AI applications
- Monitor LLM traces, latency, token usage, and model performance using Langfuse
- Integrate AI testing frameworks into CI/CD pipelines
- Analyze test results and identify model, prompt, retrieval, and application-level issues
- Collaborate with AI/ML Engineers, Data Scientists, Developers, and QA teams
Skills
- 5+ years of experience in Software/QA Testing , preferably with AI/ML applications
- Hands-on experience with Generative AI / LLM Testing
- Strong experience with DeepEval
- Hands-on experience with Langfuse
- Good knowledge of LLM evaluation metrics such as:
- Answer Relevancy
- Faithfulness
- Contextual Relevancy
- Contextual Precision
- Hallucination detection
- Toxicity
- Bias and safety
- Strong Python programming and test automation skills
- Experience testing RAG applications and LLM pipelines
- Knowledge of OpenAI/LLM APIs, embeddings, vector databases, and prompt engineering
- Experience with Pytest or similar Python testing frameworks
- Knowledge of API testing using REST APIs/Postman
- Understanding of CI/CD and Git
- Experience with LangChain / LlamaIndex
- Experience with LLM-as-a-Judge evaluation
- Knowledge of AI Red Teaming / Responsible AI testing
- Experience with Selenium/Playwright for AI application UI testing
- Experience with AWS/Azure/GCP AI services
- Experience integrating DeepEval/Langfuse into CI/CD pipelines
Company Overview