Senior Site Reliability Engineer, Tooling & Automation

Senior Site Reliability Engineer, Tooling & Automation United States, Remote

Job Description

We are looking for a Senior Site Reliability Engineer (SRE) who builds software. You will join our SRE organization and build the tooling that our SRE team and the wider operations organization run on. You will also help us take Hydrolix into air-gapped and other restricted customer environments, where we cannot reach the cluster and the customer cannot reach us. This is a highly technical, hands-on role. We are hiring for engineering depth: you design, build, and own production systems, and you prefer to automate a problem away rather than handle it by hand.

Key Responsibilities

Tooling and Automation: Design, build, and maintain production-grade tooling used daily by SRE and operations teams, including fleet management, automation APIs, dashboards, and ChatOps integrations. Replace repetitive operational work with automated systems that make decisions, not scripts that notify a human. Measure the toil you remove. Air-Gapped Deployments: Build deployment, upgrade, and lifecycle tooling for Hydrolix in air-gapped and disconnected environments, including offline artifact and image distribution and update and license flows. Supportability in Restricted Environments: Build diagnostics, support bundle, and offline telemetry tooling so we can troubleshoot and support clusters we cannot access directly. Infrastructure Reliability and Optimization: Deploy, maintain, and ensure a highly reliable fleet of Kubernetes clusters and Hydrolix deployments across multiple cloud platforms. Design, implement, and maintain systems and processes to enhance the reliability, availability, and performance of our services. CI/CD Management: Build and optimize CI/CD tools and processes to ensure efficient and reliable deployments. Monitoring and Incident Response: Develop and manage monitoring, alerting, and incident response strategies to minimize downtime and enable rapid recovery. Root Cause Analysis: Conduct comprehensive root cause analyses for system failures, implementing long-term preventive measures in code. On-Call Support: Participate in covering weekday business hours and once-monthly weekend shifts. Collaboration and Customer Engagement Cross-Functional Teamwork: Work closely with software engineering, infrastructure, product, and support teams to integrate reliability practices into every stage of the development lifecycle. Reliability Advocacy: Champion SRE best practices and foster a culture of operational excellence across the organization. Global Team Collaboration: Collaborate with a distributed team of engineers worldwide to provide round-the-clock support. Customer Support: Work with support teams and customers to resolve reported incidents and deployment issues, including in environments with limited or no connectivity. Qualifications and Skills Experience: 5+ years of experience. SRE Expertise: Proven experience as a Site Reliability Engineer or similar role, with a history of supporting complex distributed systems (minimum five years). Software Engineering: Production experience building systems in Go, Python, or Rust. We want to hear about services, APIs, controllers, or platforms you designed and owned, not only scripts. Kubernetes and Cloud Platforms: Proficiency with Kubernetes and at least one major cloud platform (AWS, GCP, Azure, or Linode). Observability Tools: Experience with monitoring and debugging tools like Prometheus, Vector, Grafana, Superset, or Kibana. Database Knowledge: Experience with SQL databases; familiarity with PostgreSQL is a plus but not required. Linux Expertise: Strong experience with Linux systems, including performance tuning and system-level troubleshooting. Communication Skills: Excellent written and verbal communication skills, with the ability to convey technical concepts clearly to diverse audiences, including customers and cross-functional teams. Plus: Experience deploying or supporting software in air-gapped, on-premises, or other restricted-network environments.