Senior Software Engineer - AI Infrastructure

4w1 month ago

The Allen Institute for AI

Seattle, US · Full-time · $126,000 – $189,000

About this role

You are an expert systems engineer who occupies the space between high-level software orchestration and low-level system performance. You are as comfortable designing a resource allocation algorithm in Go as you are debugging a NCCL timeout, motivated by the idea that world-class infrastructure should be a catalyst for public good.

Your challenge is to build the infrastructure that makes transparent AI breakthroughs possible, bridging the gap between researchers and massive GPU clusters. You will design and scale our orchestration layer to ensure the highest value workloads receive GPU time, moving HPC operations from manual intervention to high-level automation.

Ai2 remains a lighthouse for Open Science, a non-profit research institute dedicated to building AI for the common good. We operate at the pace and scale of a world-class tech startup but with the intellectual soul of a research lab, prioritizing technical elegance and open-source contributions over quarterly profit targets.

You will be a senior technical contributor responsible for ensuring that when a researcher submits a job, the software schedules it intelligently and the hardware executes it flawlessly. This role offers the unique freedom to pursue long-term stability and scientific impact rather than proprietary secrecy.

Requirements

Expertise in high-level software orchestration and low-level system performance
Proficiency in designing resource allocation algorithms in Go
Experience debugging NCCL timeouts and other HPC-level issues
Ability to blend the rigor of a Senior Software Engineer with the pragmatic, hands-on urgency of an HPC operator
Comfortable building systems that thrive under the pressure of training world-class AI models
Experience with large-scale GPU cluster coordination and job scheduling

Responsibilities

Design and scale the orchestration layer to ensure highest-value workloads receive GPU time
Move HPC operations from manual intervention to high-level automation
Work directly with researchers to squeeze every bit of performance out of the GPU-accelerated computing environment
Independently design and deliver critical systems spanning the entire stack—from the Beaker job scheduler to the execution runtime
Build innovative tooling and software-defined infrastructure to accelerate researcher velocity and automate cluster health management
Ensure that when a researcher submits a job, the software schedules it intelligently and the hardware executes it flawlessly

Benefits

Base salary range of $126,000 - $189,000
Generous bonus plans to provide a competitive compensation package
Radical transparency: we release the data, the training code, and the infrastructure insights
Mission-driven work focused on scientific impact rather than quarterly profit targets
On-site work arrangements available with flexibility based on position and team