Spotify · Primly Community

Spotify data engineer interview: pipelines, SQL, and what actually trips people up

analyst_ana · 4 replies

did the Spotify DE loop about two months ago for a senior data engineer role. sharing this because the DE-specific experience is almost invisible online compared to the SWE and DS loops.

the full structure: OA (take-home coding, 2 problems, 2 hours) technical phone screen (60 min) final round: 4 interviews over 2 days (virtual)

the OA the first problem was a SQL problem, moderately complex, the kind where you're aggregating event data with multiple dimensions. the second was a Python problem about processing a stream of records with some transformation logic. both felt like 'can you write clean production-like code' more than algorithmic puzzles. no trick DP or graph stuff.

technical phone screen one interviewer, 60 minutes. mix of: a coding problem (similar to OA level) questions about pipeline design: 'how would you design a pipeline to handle late-arriving events at scale' Kafka came up. i mentioned i'd worked with it and they went deeper. if you've used Kafka, study your consumer group semantics and offset management. if you haven't, know the concepts.

final round

pipeline design round: they gave a scenario roughly like: you need to ingest raw streaming event data from multiple sources, normalize it, and make it available for real-time dashboards and batch ML training jobs. classic lambda vs kappa debate, and they explicitly asked me to name the tradeoffs.

SQL round: harder than the OA, closer to the DS SQL round ds_dmitri described in the other thread. window functions, time-series aggregations, one question about deduplication strategies for streaming events with at-least-once delivery.

behavioral: standard values questions. one specific one: 'tell me about a time a pipeline you owned caused a downstream data quality issue. what happened and what changed after.'

what trips people up late-arriving events. everyone can design a happy-path pipeline. the Spotify DE interview specifically probes what you do when an event arrives 6 hours late and your downstream tables are already computed. know your options (reprocessing, flagging, watermarking) and have opinions.

4 replies

consultant_cam

the late-arriving events question is so real. every streaming system eventually gets bitten by this. what was the approach you outlined that they responded well to?

alex_design

i talked about watermarking for the streaming layer (basically defining a late event window tolerance) plus a reprocessing job for events that fall outside it, with lineage tracking so downstream consumers know when a table has been recomputed. they seemed more interested in the 'how do your consumers know something changed' part than the ingestion mechanics.

ux_uma

glad you mentioned the SQL overlap with the DS loop. it confirms the bar is similar. the dedup strategies question is one i got too, framed slightly differently (find duplicate events in a table and explain why they might exist).

sec_sasha

the lambda vs kappa architecture debate is one of those questions where there's no wrong answer but there's definitely a wrong way to answer: picking one without acknowledging the tradeoffs. kappa is cleaner but operationally harder for replay-heavy use cases. lambda has two codebases to maintain. both are fine, just know why you're choosing.