[Remote] Software Engineer, Data Infrastructure
Note: The job is a remote job and is open to candidates in USA. Cohere is a security-first enterprise AI company that builds foundation AI models and end-to-end products for real-world business applications. The Data Infrastructure team is seeking a Software Engineer to build and operate a distributed storage system supporting large-scale model training and evaluation, including data movement across regions and backends.
Responsibilities
- Design, build, and operate the distributed storage system that feeds model training and evaluation
- Run this system multiple on Kubernetes clusters at petabyte scale
- Work with researchers and training-infra teams on how jobs actually read and write data, and turn that into throughput, latency, and durability requirements
- Work through the networking, I/O, and consistency problems of moving large datasets and checkpoints across regions and backends, with GPU idle time and time-to-insight as the measures of success
Skills
- Strong storage fundamentals, including replication, consistency, caching, and data lifecycle management
- Strong coding ability. We work in Python and Go; experience in either is enough, but you should be willing to pick up the other
- Experience running stateful systems on Kubernetes, including Persistent Volumes, CSI drivers, and StatefulSets
- Hands-on experience with cloud object storage such as S3 as well as POSIX-style filesystems
- Experience with parallel or HPC filesystems such as Weka, VAST, or Lustre
- Familiarity with the data-loading and checkpointing patterns used in large-scale model training
- An interest in how large models are trained and evaluated, including the trajectory and eval data those runs generate, and a desire to understand the workloads the platform is serving
Benefits
- Offers Equity
- A weekly lunch stipend of $75/£75 or equivalent in your local currency for lunch.
- Full health and dental benefits, including a separate budget for mental health.
- RRSP matching, 401K, Pension Scheme.
- 100% Parental Leave top-up for up to 6 months, for either parent.
- Annual enrichment benefits: Arts & culture, fitness/wellness, quality time, and a workspace improvement credit.
- Education & learning stipend for conferences, courses, and coaching.
- 6 weeks of paid vacation (30 working days!)
- Budget for traveling to other offices if you are remote, plus an annual company offsite.
- Cohere is remote-friendly.
- For those in the office: a daily lunch program, plenty of snacks, and regular community and social events.
- For those not near an office: a co-working benefit so you can work alongside others in your city.
- Everyone receives a $500 home office stipend to set up your workspace properly.
Company Overview
Company H1B Sponsorship