[Remote] AI Application Engineer
Note: The job is a remote job and is open to candidates in USA. Software Guidance & Assistance, Inc. (SGA) is seeking an AI Application Engineer for a contract assignment with an infrastructure services client. The role designs, tests, optimizes, and supports AI-powered agent experiences across customer and internal business workflows, including prompt engineering, agent orchestration, quality assurance, evaluation frameworks, and AI observability.
Responsibilities
- Design, develop, and iterate on AI agent logic for business use cases
- Create, test, and optimize prompts, system instructions, tools, and workflows to improve agent performance
- Conduct prompt experiments and A/B testing to measure effectiveness
- Analyze agent responses and refine prompting strategies for accuracy, consistency, groundedness, and user satisfaction
- Document prompt patterns, reasoning frameworks, reusable templates, and agent design decisions
- Collaborate with engineering teams to deploy prompt and agent updates through established SDLC processes
- Design and execute comprehensive test plans covering AI agents, web applications, APIs, and backend services
- Develop evaluation frameworks and scoring methodologies for AI output quality
- Create test cases covering: Accuracy; Hallucination detection; Safety; Tone and style adherence; Instruction following; Context retention; Response consistency
- Perform regression testing for prompt and model changes
- Track, prioritize, validate, and retest defects using Azure DevOps
- Utilize Arize or similar observability platforms to monitor model and agent performance
- Analyze user interactions and evaluation results to identify trends and improvement opportunities
- Develop quality metrics and dashboards for: Accuracy; Relevance; Latency; User satisfaction; Defect rates; Prompt performance
- Investigate production issues and recommend prompt, workflow, or architectural improvements
- Support integration of AI capabilities into enterprise applications and workflows
- Collaborate with developers to validate API integrations, retrieval pipelines, and agent workflows
- Assist in CI/CD deployment and release validation activities using Azure DevOps
- Create and maintain automated testing assets where appropriate
- Maintain documentation for prompts, evaluation methodologies, testing procedures, and agent configurations
- Participate in sprint planning, backlog refinement, and Agile ceremonies
- Communicate findings and recommendations to technical and non-technical stakeholders
- Share best practices and lessons learned across the team
Skills
- Bachelor's degree in Computer Science, Engineering, Information Systems, or related field
- 5-10 years of experience in software engineering, QA engineering, automation testing, business systems analysis, or related technical roles
- 1+ years of hands-on experience with Generative AI, LLMs, prompt engineering, or conversational AI solutions
- Experience working with Azure DevOps or similar Agile delivery platforms
- Experience designing and executing structured test plans
- Strong analytical and troubleshooting skills
- Experience working with APIs and JSON data structures
- Excellent written and verbal communication skills
- Prompt Engineering
- Agent Design
- AI Evaluation Frameworks
- RAG Concepts
- LLM Testing
- AI Quality Assurance
- AI Observability
- Prompt Optimization
- Arize AI
- Azure DevOps
- Azure OpenAI
- Git
- Postman
- Python
- SQL
- REST APIs
- JSON
- Test Automation Frameworks
- Experience with Arize AI or comparable AI observability tools
- Experience with Azure OpenAI, OpenAI, Anthropic, Gemini, or other LLM platforms
- Knowledge of Retrieval-Augmented Generation (RAG) architectures
- Experience evaluating AI systems using human and automated scoring methods
- Familiarity with vector databases and semantic search
- Experience creating automated test frameworks
- Understanding of responsible AI principles and AI governance
- Experience with Python, SQL, JavaScript, or C#
- Knowledge of prompt versioning and experimentation practices
- Jira (optional)
- Power BI (optional)
Benefits
- This role will be remote.
Company Overview
Company H1B Sponsorship