[Remote] Principal AI Infrastructure Engineer
Note: The job is a remote job and is open to candidates in USA. Highrise AI is seeking a hands-on AI infrastructure engineer to bring large-scale GPU clusters from installation through production readiness and customer handoff. The role focuses on cluster integration, bare-metal provisioning, performance validation and optimization, and Tier 3 support for operating clusters.
Responsibilities
- Lead technical bring-up of materially complete, racked, powered, and cabled AI clusters through production readiness and customer handoff
- Establish and validate system baselines across BIOS/BMC, firmware, GPU drivers, operating systems, and cluster software stacks
- Deploy and optimize automated bare-metal provisioning, imaging, node configuration, and lifecycle workflows at scale
- Configure and troubleshoot advanced Linux and GPU systems, including kernel, driver, accelerator, and host-level performance settings
- Integrate and validate Kubernetes and Slurm environments in partnership with platform, networking, and operations engineers
- Execute NVIDIA Field Diagnostics, L11/L12 validation, burn-in, stability, and hardware health qualification procedures
- Run and analyze NCCL collective communication tests, GPU benchmarks, and distributed workload performance at node, rack, and cluster scale
- Troubleshoot GPU, NVLink/NVSwitch, PCIe, RDMA, and InfiniBand/Ethernet performance issues, partnering with network engineers on fabric-related remediation
- Identify configuration bottlenecks and optimize throughput, latency, scaling efficiency, and cluster reliability
- Own technical acceptance criteria, test evidence, defect tracking, remediation verification, and readiness recommendations for commercial service
- Serve as a senior escalation point for complex production incidents involving compute nodes, GPU software, provisioning, scheduling, and degraded performance
- Perform deep root cause analysis, coordinate vendor engineering escalations, and lead corrective actions for recurring or systemic issues
- Support fleet-level upgrades, configuration changes, cluster expansions, and post-maintenance validation with appropriate change controls
- Develop diagnostic tooling, automation, runbooks, and operational handoff material for SRE, NOC, and site support teams
- Work closely with infrastructure engineering, site operations, OEMs, and software/platform teams to resolve cross-stack issues and meet delivery milestones
- Provide concise technical status, benchmark findings, risks, and acceptance recommendations to engineering leadership and stakeholders
Skills
- Bachelor's degree in computer science, engineering, or a related field, or equivalent practical experience
- 8+ years of hands-on Linux systems engineering, HPC, cloud, or large-scale infrastructure experience, including senior-level troubleshooting
- Proven experience bringing GPU or HPC clusters into production and qualifying systems against performance and reliability targets
- Strong expertise in Linux, automation (Python/Bash/Ansible), bare-metal provisioning, configuration management, and systems diagnostics
- Hands-on experience with NVIDIA GPU platforms, NCCL, Field Diagnostics, and multi-node performance testing
- Practical experience with Kubernetes and Slurm; familiarity with high-speed Ethernet, InfiniBand/RDMA, and parallel or distributed storage
- Deep technical curiosity and the ability to isolate difficult problems across hardware, firmware, operating systems, and distributed workloads
- Strong ownership of outcomes, with a systematic approach to testing, documentation, root cause analysis, and continuous improvement
- Comfortable collaborating with highly experienced peer engineers while driving independent technical decisions and execution
- Clear communication with engineering leadership, vendors, site personnel, and customer-facing stakeholders
Benefits
- Hybrid or remote, depending on location.
- Work with bright, driven peers from a range of educational and professional backgrounds.
- Design and pitch new products, services, and other initiatives to the leadership team.
- Debate ideas and alternatives in a meritocratic setting, with lessons from both senior and junior team members.
- Build a lifelong network of friends and professional connections at the cutting-edge intersection of technology, energy, and infrastructure.
Company Overview