Senior Site Reliability Engineer

Il y a 9 heures

Brussels, Belgique REALM Temps plein
h3Senior Site Reliability Engineer, GPU Infrastructure /h3 pRemote, EMEA or APAC | Full-time | Competitive base + meaningful equity /p h3The company /h3 pI’m hiring for an AI infrastructure company that sources GPU compute from dozens of providers and runs it as one managed platform for AI labs and model builders. They have tens of thousands of GPUs under management and a team of around 50. They’re profitable and backed by some of the best-known names in AI. It’s a confidential search, so I can’t name them here, but I’ll tell you everything on a call. /p h3The role /h3 pThis isn’t a generalist SRE seat. You’ll run large GPU clusters for distributed training and inference, spread across multiple providers and regions, and work directly with the customers pushing them hardest. When a training run slows down, you’ll be the one tracing it from the network fabric through the kernel to the framework. /p h3What you’ll own /h3 ul liDesigning and running multi-provider, multi-region GPU clusters for large training jobs /li liActing as the technical contact for customers, from onboarding through to live troubleshooting and tuning /li liSLOs, capacity planning and reliability across a mixed GPU fleet /li liKeeping InfiniBand, RoCE and NVLink fabrics healthy /li liMonitoring that shows what’s happening with GPU utilisation, interconnect throughput and hardware health /li liAutomation for provisioning, health checks, self-healing and driver and firmware updates /li liLeading incidents where the root cause could sit in hardware, network, orchestration or the ML stack /li /ul h3What you’ll need /h3 ul liYou’ve run large GPU clusters (A100, H100, B200 or similar) in production yourself /li liProduction experience with InfiniBand, RoCE or NVLink for distributed training /li liA good understanding of how big training jobs behave at the systems level: NCCL, CUDA, PyTorch distributed or similar /li liStrong Linux, including kernel tuning and NVIDIA driver and CUDA management /li liKubernetes with GPU workloads, or Slurm and other HPC schedulers (either counts) /li liPython, Go or Bash for building proper tooling, plus Terraform, Helm, Ansible or similar /li liGPU monitoring with DCGM and nvidia-smi /li liExperience leading messy incidents where the cause could be anywhere from firmware to application code /li /ul h3Nice to have /h3 ul liParallel file systems like VAST, WEKA, Lustre or GPFS /li liProfiling and tuning distributed training performance /li liPhysical cluster design: racks, power, cooling, network topology and burn-in /li /ul