DS roles at AI labs are interesting because the interview usually mixes standard DS components with questions that are actually specific to evaluating AI systems. Anthropic was no exception. Here's how the loop broke down for me.
Role: Data Scientist on a team focused on model evaluation / applied measurement. Not a pure ML research role, not a pure analytics role.
SQL round: This was a live coding session in a shared editor. Two questions. First was medium difficulty: multi-table join, aggregation, a window function. Second was harder: something involving ranking across subgroups with a condition that made the naive approach wrong. I use SQL daily so this was fine. If you're rusty, actually practice writing queries from scratch, not just reading them.
Statistics / experiment design: This was the most interesting round. They asked me to design an experiment to evaluate whether a change to a model improved user outcomes. The catch: the outcome variable is hard to define and measure. So a lot of the round was about how I'd operationalize the metric, what the risks of a bad metric are, and how I'd handle the fact that model outputs are correlated in ways that violate standard A/B test assumptions.
Basic stats was a given but the real test was applied judgment about measurement, not textbook hypothesis testing.
Case / product sense for DS: Given a scenario about model behavior at scale, how do I prioritize what to investigate. They wanted to see structured thinking and an ability to generate hypotheses quickly.
No take-home in my loop. I've heard some candidates get one and some don't. Not sure what drives that.
Overall difficulty: Harder than standard DS loops at analytics-heavy companies. Lighter than a full ML research loop. The experiment design round was the differentiator; I'd put at least half your prep time there.
Comp: I didn't get an offer so I don't have a final number. The recruiter mentioned the DS band at senior level was in the $280k-$360k TC range.