Did the Cohere system design loop last month for a senior (roughly L5 equivalent) backend role. Coming from a big tech background I was curious how different it would feel. Short answer: quite different, in mostly good ways.
What they actually asked
The prompt was something like: design a system to handle real-time completions requests for an enterprise customer with specific latency SLAs (they gave a rough number, something under 200ms p95). You're supposed to figure out how to handle bursting load, queue management, graceful degradation when models are overloaded.
This is meaningfully different from the classic "design Twitter/Uber" style prompts. They expect you to reason about ML model serving specifically. Knowing concepts like: batching and its tradeoff with latency async vs. sync inference caching strategies for deterministic prompts GPU memory constraints at a conceptual level
...will serve you well. You don't have to be an ML researcher. You just have to have thought about this class of systems before.
How deep they go
At the senior level they pushed into operational stuff. How do you monitor this? What breaks first under load? How do you handle a single-tenant customer whose traffic spikes 10x? I spent a good 20 minutes on failure modes and monitoring, which was closer to a proper SRE conversation than a design exercise.
How it differs from FAANG
At big tech, system design is often more about demonstrating you know the canonical patterns (consistent hashing, Kafka for streaming, sharded DBs). Cohere felt more exploratory, more like they wanted to see your reasoning process and less like they were checking if you'd memorized the standard playbook. The interviewer pushed back on my assumptions, but in a collaborative way.
No whiteboard. Screen share on Excalidraw or whatever you prefer. The interviewer was actively participating, not just watching.
Level: senior IC, about 9 YOE, distributed systems background.