[Remote] LLM DevOps / Inference Engineer
Note: The job is a remote job and is open to candidates in USA. Xaxis Solutions is seeking an LLM DevOps / Inference Engineer to support a healthcare-focused AI benchmark and evaluation platform. The role owns infrastructure for the benchmark harness, self-hosted model serving stack, and platform reliability, including provisioning GPU infrastructure, optimizing model inference, and managing AWS cloud systems.
Responsibilities
- Build and maintain AWS infrastructure using Terraform, including networking, compute, orchestration, secrets, observability, and CI/CD
- Deploy and manage self-hosted LLM inference workloads on GPU infrastructure
- Implement batching, quantization, autoscaling, and multi-GPU strategies for efficient inference
- Develop standardized processes for onboarding new models quickly
- Build provider abstraction layers for API-based models such as OpenAI, Anthropic, and Gemini, as well as self-hosted models
- Implement rate limiting, retries, backoff, quota management, request/response logging, and cost attribution
- Ensure benchmark runs are reproducible and cost-efficient through pinned model/container versions and optimized cloud capacity
- Build observability dashboards covering throughput, latency, token usage, GPU-hour costs, failures, and per-model performance
- Support burst workloads while preventing orphaned or unnecessary cloud resources
- Contribute to Trusted Execution Environment (TEE) architecture and evaluate AWS Nitro Enclaves and comparable confidential-computing solutions
Skills
- Hands-on experience launching and managing inference containers
- Strong experience building and maintaining AWS infrastructure as code, including networking, compute, orchestration, secrets, observability, and CI/CD
- Experience working in a startup or fast-paced environment
- Strong AI/LLM and model infrastructure experience
- Excellent communication skills
- US Citizens will be given first preference
- Active and complete LinkedIn profile
- Production-scale AWS infrastructure experience
- Strong knowledge of EKS/ECS, EC2 GPU instances (G5/G6, P4d/P5), VPC, IAM, KMS, Secrets Manager, ECR, CloudWatch, and Service Quotas
- Strong Terraform / Infrastructure as Code experience
- Hands-on experience with LLM model serving and inference optimization
- Practical knowledge of batching strategies, KV cache behavior, quantization, and multi-GPU sharding
- Experience managing GPU workloads with EKS/ECS, node autoscaling, and CUDA-based container pipelines
- Strong reliability and cost-engineering experience, including SLOs, monitoring, alerting, and cloud cost optimization
- Experience with AWS SageMaker and Bedrock
- Strong Python development skills
- Knowledge of confidential computing, including Enclaves, remote attestation, and sealed key release
- Understanding of healthcare compliance, including HIPAA-eligible services, BAA requirements, audit logging, and PHI access controls
- Experience with security hardening, image scanning, and network egress controls
Company Overview