did the NVIDIA MLE loop for a role on their model training infrastructure team. want to give an honest breakdown because the job posting made it sound like you need a PhD in numerical methods and 5 years of CUDA kernel tuning. reality was more nuanced.
coding: 2 rounds. one leetcode-style (I got a graph problem, medium-hard, nothing specifically ML). one applied: given a mock training run log, find the bottleneck. I wrote Python to parse the log, identify GPU idle time, and suggest what part of the pipeline was the bottleneck. this is the kind of thing you actually do as an MLE so it felt fair.
ML systems design: this was the heavy round. design a distributed training system that can handle LLM pretraining at scale. I talked through: data parallelism vs. model parallelism vs. pipeline parallelism (tensor parallel specifically), gradient checkpointing tradeoffs, communication primitives (all-reduce, ring vs. tree), checkpointing strategy, and fault tolerance. they pushed on failure modes. what happens when one GPU in a 128-GPU run fails at hour 16? not a theoretical question for this team.
ML depth: 30 min focused on what you know cold. I got questions on transformer architecture internals (attention complexity, KV cache), quantization (INT8/FP8 and when precision degrades), and mixed precision training. they wanted to know if I'd actually trained large models or just read about it.
behavioral: 3 Qs. 'tell me about a model that didn't work and what you did', 'tell me about optimizing something that nobody else thought was a priority', 'tell me about working with research vs. production tensions.' that last one comes up a lot in research-adjacent MLE roles.
on CUDA: you don't need to write kernels for the loop. you DO need to understand what kernels are doing and where the performance ceiling is. if you can't explain why a fused attention kernel is faster than naive attention, you're going to struggle in the ML depth round.
my offer was in the Santa Clara band. base around $210k at their equivalent of senior IC, RSUs were the real number. total comp first year around $380k with signing. that's real not inflated but I'll just say it was more than I expected given how the job posting described it.