GitHub · Primly Community

GitHub data engineer interview, pipelines and SQL, here's what actually came up

de_derek · 4 replies

Finished a GitHub DE loop last quarter. This is for a data engineering role on their data platform team, not a software engineering role. The process and content are different and there's almost nothing written about it, so here's the full breakdown.

Recruiter screen. Standard. They asked about background with dbt, Spark, and Airflow specifically. Those three came up in the job description and they were checking for real familiarity, not just keyword matching. I'd used dbt and Airflow heavily, Spark less so, and being honest about the depth helped.

Technical phone screen. Two parts: SQL and a pipeline design discussion.

The SQL was harder than I expected. Not just joins and aggregations. I got a window functions problem involving calculating user engagement metrics over rolling 7-day periods across billions of hypothetical rows. They wanted me to explain how I'd handle performance at GitHub's scale. Think: partitioning, materialization strategy, not just query correctness.

The pipeline design question was about ingesting webhook event data at high volume. How do you handle schema evolution, late-arriving events, exactly-once semantics. Very GitHub-specific, their webhook firehose is a real engineering challenge.

Onsite (virtual). Four rounds for me: SQL and data modeling deep dive Distributed systems for data (how do you partition large repo activity datasets, handle backfill jobs without impacting prod) Behavioral (async-first theme, same as the SWE loop) Hiring manager conversation

The data modeling round caught me off-prepared. I knew how to write SQL but they wanted me to design a schema that could answer a whole class of analytical questions efficiently. Practice designing dimensional models, not just tables.

Tech stack they mentioned using: Spark, dbt, Airflow, BigQuery or similar OLAP store, Kafka for event streaming. Knowing the whole pipeline not just the query layer mattered.

Total timeline from application to offer was about 6 weeks. Debrief took 7 business days, which felt long but they were apparently running multiple DE candidates in parallel.

Comp offer for senior DE remote was around 210k total with equity refresh. I didn't take it but thought it was fair for the role.

4 replies

ds_dmitri

The rolling window functions at scale question is a classic. The answer they want is usually: don't compute in the query, precompute and materialize incrementally. Did you end up going that direction?

de_derek

Yeah, I talked through materializing daily user activity snapshots and then using a simple join rather than a true window over raw events. Interviewer seemed happy with that direction.

analyst_ana

The schema evolution question for webhook ingestion is something I've dealt with in my current role and it's genuinely hard. Did they get into how you handle breaking changes vs additive changes differently?

backend_bekah

The Kafka mention for event streaming makes sense. GitHub's webhook volume is enormous. Any company at that scale needs a proper event bus, not just firing HTTP calls and hoping they land.