GitHub · Primly Community

GitHub machine learning engineer interview, what actually happened in my loop

ml_mike · 4 replies

went through GitHub's MLE interview process for a role on one of their AI teams, Copilot adjacent. i have 8 years in ML so this was for a senior IC position. sharing because i couldn't find much specific GitHub MLE interview info when i was prepping.

context: GitHub has expanded their ML org significantly since Copilot. there are teams working on code generation models, code review suggestions, vulnerability detection, and some search/ranking stuff. the role i was interviewing for was on the ML platform side, not pure research.

the loop (remote, 2026):

recruiter screen: standard, 30 min. they specifically asked about my experience with LLM fine-tuning and evaluation. this is not a 'classical ML' shop at this point. if your background is all tabular data and tree ensembles, you'll need to bridge that gap.

ML system design (75 min): design a system that flags potentially buggy code commits using ML signals. this was genuinely meaty. they wanted: how do you define the ML problem, what's your training data strategy, what features make sense for code quality, how do you handle model staleness as coding patterns evolve, and how do you measure success without ground-truth labels. we spent probably 20 minutes on the evaluation/measurement problem alone.

coding round (60 min): one medium-ish problem, not LeetCode style, more like 'implement a sliding window evaluation metric for streaming model predictions.' python. focused on efficiency but not tricks.

ML depth / paper discussion: a researcher-type interviewer who wanted to dig into something from my resume. we ended up talking about fine-tuning strategies and catastrophic forgetting. they asked me to walk through a tradeoff between LoRA and full fine-tune for a code LLM.

behavioral / cross-functional: how have you collaborated with product, how do you handle pushback when your model recommendations conflict with what eng wants to ship.

what matters: GitHub MLE interviews are about applied ML judgment, not research chops. the coding bar is real but not the main event. the system design and ML depth rounds are where you either differentiate or get filtered. know your eval frameworks, know how to reason about data quality for code-domain training data, and have opinions about LLM evaluation that go beyond perplexity.

4 replies

ds_dmitri

the 'how do you measure success without ground-truth labels' question is one of the hardest in applied ML interviews and most people bomb it by immediately jumping to proxy metrics without acknowledging why they're proxies. did you go down the human eval / A/B test path, or something else?

ml_mike

both. i talked about using downstream signals (did the developer revert the commit, did a bug report get filed within X days) as delayed supervision, plus a human eval process for a held-out set of commits. interviewer pushed on the latency of the delayed labels and we talked about online learning vs batch retraining. that's where the round got interesting.

sec_sasha

curious about the vulnerability detection angle. did they talk about any of that in the cross-functional round or was it purely the role you interviewed for? asking because that intersection of ML and AppSec is where i'd want to go.

laidoff_lena

the LoRA vs full fine-tune tradeoff question is interesting to see in an interview context. that means they're actually thinking about training costs at inference, not just throwing compute at it. seems like a shop that cares about practical ML, not just benchmarks. good sign.