Career Switchers · Primly Community

data scientist to machine learning engineer: the skill gap is real and here's how to close it

ml_mike · 5 replies

i see this question a lot and most answers either minimize the gap ('just learn MLOps!') or overstate it ('you have to basically retrain as a software engineer'). after doing this switch myself and then watching several colleagues try it, here's a more honest take.

what you have that translates: you understand why models make the choices they make. most SWEs don't. you can read papers and understand what's being proposed. this matters more at ML-heavy companies than people realize. you've probably debugged training data issues, which is 60% of ML eng work.

what you're probably missing: production-grade Python. notebooks are not production code. dependency management, typing, testing, logging, error handling: these matter and you'll need to ship them. distributed systems basics. if your company runs training at scale, you'll need to understand something about how spark or ray or kubernetes works, even if you're not building the infrastructure. ML platform fluency. SageMaker, Vertex AI, internal platforms: knowing how to actually get a model from a notebook to a running endpoint, with monitoring and rollback. software design instincts. ML engineers are still engineers. you'll be code-reviewed by people who will flag the things a data scientist would call 'good enough.'

how i closed the gap: took about 8 months while employed. built two side projects that required training a model, serving it, and monitoring its performance in a real deployed environment. not toys. actual apps i left running. learned to hate myself for the things i wrote in month 1 and rewrote them in month 6. that iteration is the point.

the leveling hit: i went from a mid-level DS to a junior MLE. took about a year to get back to mid. if you do it right, the ceiling as a senior MLE is higher than as a senior DS at most companies, so it's worth it if the work is what you want to do.

5 replies

ds_dmitri

the production Python point is the one that slapped me when i made this switch. i genuinely thought i knew Python. i did not know Python. 'i can make a notebook run end to end' and 'i can write a service that other engineers don't silently dread touching' are different skills.

de_derek

as a data engineer who works next to both DS and MLE: the people who make the switch fastest are the ones who already had opinions about code quality. if you've been writing 800-line notebooks with no functions and no documentation, the learning curve is steep. if you already organize your work, it's manageable.

ml_mike

this is a better diagnostic than i gave. 'do you have opinions about your own code' is the real question.

bootcamp_bri

is this path more realistic than the reverse? i'm a SWE trying to move into ML and wondering which direction is harder.

ml_mike

bri: i'd say SWE to MLE is harder in the domain-knowledge direction (you need to actually understand what a model is doing) and DS to MLE is harder in the engineering direction. both paths work. which gap is easier for you to close depends on where you started.