Perplexity · Primly Community

Perplexity data scientist interview (SQL + case + stats), what each round actually looks like

analyst_ana (Primly starter) · 6 replies

Went through the Perplexity DS loop in February 2026. Senior DS role. Sharing this because I couldn't find anything specific about their DS process before interviewing.

Recruiter screen: Standard. They asked about my stack (SQL, Python, experiment design) and why I wanted to do DS at an AI search company. I mentioned I was interested in the challenge of measuring quality in generative outputs, which seemed to resonate.

Technical screen (45 min with a DS on the team): Two SQL problems. The first was a window function / ranking question (find the Nth highest value per group, classic). Second was messier: multi-table join with some ambiguous logic that required me to ask clarifying questions about how the tables related. The second problem was clearly more important. They watched how I handled ambiguity.

After SQL, we spent 10 minutes on stats. The question was: you're running an A/B test on answer quality. Users in variant B rate answers higher. Walk me through how you'd decide if this is real. I talked about sample size, test duration, novelty effect controls, checking for covariate imbalance. They asked specifically about what I'd do if click-through rate and explicit rating moved in opposite directions. Good question. I said I'd treat them as separate hypotheses measuring different user behaviors and wouldn't combine them.

Case study (60 min with a PM and senior DS): Given a scenario: Perplexity launches a new feature that shows related questions at the bottom of a search result. How do you measure if it's working? No data provided, open-ended. I scoped the goal (engagement vs. satisfaction vs. long-term retention), proposed metrics, designed a rough experiment, called out the things that could make the data misleading (related questions might cannibalize the original query's value rather than add to it).

They pushed on: how do you know if a user clicking a related question is a success signal or a "they didn't find their answer" signal. That's the real complexity. I talked about session depth, return rate, and qualitative signal from follow-up queries.

What made the difference (at least I think): Treating the ambiguity as a feature, not a bug. DS at an early-ish company means the data is messy and the product questions are novel. They seemed to want someone comfortable with that, not someone who needs a clean dataset handed to them.

6 replies

analyst_ana (Primly starter)

The click on related question being a success or failure signal is such a real problem in search product analytics. Did they ever tell you how they actually think about it internally? Or was that the point, that it's genuinely open?

alex_design (Primly starter)

They didn't give me their answer. I think it was genuinely a question they think about and wanted to see how I'd approach it. The signal ambiguity in AI search is real and there's no consensus even internally at companies doing it.

de_derek (Primly starter)

Was the SQL done in a specific dialect (Postgres, BigQuery, SparkSQL)? I always wonder if I need to account for dialect differences in practice rounds.

brand_ben (Primly starter)

They said any dialect was fine and they'd ask if something was unclear. I wrote standard SQL with a few Postgres idioms and it was fine. They're not testing dialect trivia.

contractor_kai (Primly starter)

The novelty effect control question in A/B testing is something a lot of candidates skip because it's easy to forget. Good that you mentioned it. That alone probably moved the needle.

Primly Team

One stage people tend to underestimate in senior DS loops is metrics arbitration when signals disagree. In AI search contexts, it is common to see explicit quality ratings move one way while CTR, dwell time, or follow-up queries move another. A strong answer is to lay out a simple decision framework: (1) define the primary objective (user trust, task success, or retention) and which metric is the closest proxy, (2) diagnose mechanism differences (did the model become more verbose, slower, or change answer formatting), (3) run guardrail checks (latency, refusal rate, hallucination reports, safety), and (4) propose a follow-up experiment or slice to resolve the conflict (new vs power users, query type, intent).

What metric conflict did you see in your process, and how did the interviewer want you to trade off short-term engagement vs long-term quality/trust?