Stanford Research Computing seeks an exceptional GPU Cluster Lead Engineer to oversee technical operations, optimization, and strategic development of Marlowe, Stanford’s NVIDIA SuperPOD. This role combines deep technical expertise in GPU computing, large‑scale cluster management, and leadership in supporting a diverse research community. You will serve as the technical authority on GPU infrastructure, driving system performance and reliability while enabling groundbreaking research in AI/ML, computational biology, physics, and beyond.
Key Responsibilities- System Operations & Management
- Lead day‑to‑day operations of the GPU Cluster, ensuring optimal uptime and performance.
- Architect monitoring, alerting, and observability solutions using Prometheus, Grafana, DCGM, and Base Command Manager.
- Manage job scheduling and resource allocation using Slurm, implementing advanced GPU partitioning and configurations.
- Coordinate maintenance windows, system upgrades, and capacity expansions; lead incident response and root‑cause analyses.
- Manage system storage, optimization, benchmarking, and observability reporting.
- Performance Optimization & Engineering
- Design performance tuning strategies for GPU utilization, job throughput, and system efficiency.
- Optimize NVIDIA GPU fabric configurations including NVLink, NVSwitch, and InfiniBand RDMA networking.
- Develop containerization strategies using NVIDIA NGC, Docker, and Singularity/Apptainer.
- Engineer solutions for deep learning frameworks (PyTorch, TensorFlow, JAX) and CUDA application optimization.
- Benchmark system performance and collaborate with NVIDIA on optimization programs.
- User Support & Research Enablement
- Serve as primary technical consultant for researchers using GPU‑accelerated computing.
- Develop documentation, best‑practice guides, and training materials; deliver workshops on GPU computing workflows.
- Profile and optimize user workloads, scaling applications from single‑GPU to multi‑node distributed training.
- Team Leadership & Strategy
- Mentor junior engineers and contribute to strategic planning for GPU infrastructure expansion.
- Evaluate emerging GPU technologies and manage vendor relationships with NVIDIA and hardware suppliers.
- Represent SRC in ongoing interactions with the Stanford Data Sciences group on AI/ML infrastructure; participate in on‑call rotation.
- Bachelor’s degree in Computer Science, Engineering, or related field and ten years of relevant experience or a combination of education and relevant experience.
- 5+ years in HPC systems administration or research computing; 3+ years managing GPU clusters (NVIDIA A100/H100).
- Expert knowledge of NVIDIA GPU architecture, CUDA, and GPU computing principles (NVLink, MIG, GPUDirect).
- Advanced Linux administration (RHEL, Ubuntu); expertise with Slurm job scheduler.
- Experience with high‑performance networking (InfiniBand, RoCE) and parallel filesystems (Lustre, GPFS).
- Strong scripting (Python, Bash) and containerization experience (Docker, Singularity, Kubernetes).
- Familiarity with AI/ML frameworks (PyTorch, TensorFlow) and distributed training techniques.
- Experience with monitoring tools (Prometheus, Grafana) and NVIDIA DCGM.
- Experience with Base Command Manager or Bright Cluster Manager.
- Background in academic research computing or national lab environments.
- Contributions to open‑source HPC or GPU computing projects.
- Knowledge of MLOps practices and GPU virtualization (vGPU, MIG).
- Technical leadership
- Creative problem‑solving
- Excellent communication with technical and non‑technical audiences
- Strong collaboration skills
- Service‑oriented mindset
- Adaptability to rapidly evolving technology
- Work with cutting‑edge NVIDIA GPU technology enabling groundbreaking research
- Professional development opportunities
- Collaborative environment with talented engineers and researchers
- Comprehensive Stanford benefits package including health, dental, retirement, and education benefits
- Flexible work arrangements
- Constantly perform desk‑based computer tasks.
- Frequently sit, grasp lightly/fine manipulation.
- Occasionally stand/walk, writing by hand.
- Rarely use a telephone, lift/carry/push/pull objects that weigh up to 10 pounds.
- May work extended hours, evenings, and weekends.
The expected pay range for this position is $190,577 to $200,000 per annum.
EEO & Accessibility StatementStanford is an equal‑employment‑opportunity and affirmative‑action employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, protected veteran status, or any other characteristic protected by law. The University will provide reasonable accommodations to applicants and employees with disabilities. Applicants requiring a reasonable accommodation for any part of the application or hiring process should contact Stanford University Human Resources.
#J-18808-Ljbffr
Sign in
To continue your applicationor continue with email
By continuing you agree to our Terms & Privacy Policy.
