From Data Scientist to AI Engineer
The journey continues...
A little over two years ago, I wrote how I went from someone who avoided math and science at all costs, to a reasonably knowledgable and mostly competent Data Scientist who used programming, math, and statistics as part of my day to day.
I wrote this because I was proud of myself, but with full knowledge that my skillset was a work in progress. How do I know? I used the word “learn” or “learning” a full 28 times in that piece and there’s always more to know. Early on in my career I was strongly influenced by this medium piece (sorry substack), which among other things, paid homage to the 10x developer by coining the term “the infinite x data scientist”.
It is in your own, and your employers, interest that you spend a lot of time honing your skills and expanding your horizons.
Say, for example, that you face a business problem that cannot be solved with traditional ML and instead requires some custom implementation of a transformer-based architecture on multiple misaligned time series while also being interpretable. If you have never even played with simple neural networks, you will not be able to crack that problem in any reasonable amount of time. And yes, my experience is that these problems do come up, and if you do not have a solid overview of the methodologies that can solve these problems, either you will not solve the problems, or the problems will not be solved at all.
I prided myself on my versatility, so it was only natural I would heed this advice, try to keep learning, and expand the set of problems I could help out with. There were also some contextual and motivational factors that pushed me forward.
A Bifurcated Collapse of Data Science
When the Data Science monicker was first coined, Data Scientists in the popular imagination were supposed to be highly versatile, knowledgeable, and flexible coders who could not only analyze and model data, but build software around those analytics and models to produce huge ROI. However there were only so many of those types of people available, and everything got accelerated when ChatGPT launched in November 2022.
Popular buzz words like Big Data and Machine Learning were left in the rearview, and ML practitioners began pivoting from training and deploying bespoke models, to peppering products with large language model APIs to make the product more conversational or more dynamic. This was a quick pivot and at many companies it quickly de-emphasized (though not eliminated) a lot of how we used to do ML modeling work.
As a result, implementation of AI via LLMs started to look remarkably similar to other forms of backend engineering, albeit with the increased complexity that comes with working with non deterministic and highly novel LLMs. But many of the core Engineering principles were there, and thus general backend engineering chops became increasingly if you wanted to keep working in the AI space.
Data Scientists more focused on statistical modeling collapsed the other way, towards a more analytics focused and product oriented perch.
I myself wanted the opportunity to get more technical and stay as close to LLMs as I could so I embraced the challenge of leveling up my backend skills and now call myself an AI Engineer.
Ignorance as Helplessness
When my focus was around training and deploying Machine Learning models, I felt like I could solve one set of problems really effectively—prototyping and deployment—but push me off that happy path and I quickly became confused and feeling like a burden on my teammates to help unblock me.
Yes the model deployment process exposed me to some engineering fundamentals, like building data models, writing automated tests for my code, and the CI/CD process. But I didn’t know how to do so many other things like provision my own database (Terraform), debug issues with a deployment (kubernetes), or just write well crafted software that didn’t directly relate to serving a model output over an API. This bothered me.
I looked at some of my teammates who were supporting me by doing ML Ops work, and felt like I was on the wrong side of the 80/20 rule. That I could do a subset of problems better than them, but the aperture they could cover in the problem space was significantly wider than what I was capable of. I badly wanted to remedy this situation and started pushing for more general purpose engineering opportunities.
The LLM boom presented that opportunity because it created more demand within our team for that skillset. I raised my hand and never looked back. For me gaining the ability to start a project from zero and build has given me a ton of confidence and satisfaction. Not only is getting to create something you start a lot of fun, but I finally feel like I belong in tech, whereas before I felt intimidated by my lack of technical rigor and wasn’t sure if I could ever get over the hump.
When do you know how to code?
In the late 2000s/early 2010s coding bootcamps sprang up and President Obama encouraged the country’s youth to “learn to code.” I found this call to action somewhat mysterious. When I was in 5th grade I learned to design a static personal website using HTML and CSS. It was great fun, but I did not feel like I knew how to code. In fact, the more research I did the more my head spun as I learned about the wide variety of technologies available to the modern developer.
The languages: Python, java, javascript, rust, go, ruby…
The frameworks: Fast API, Djano, React, Angular, Vue, Node…
The flavors of SQL where I started as an analyst…
Testing, monitoring, and alerting
And even if I got through all of that, most online courses didn’t talk at all about infrastructure or how to actually deploy and maintain an application. That seemed to require taking yet more online courses on the many different product offerings from the cloud providers.
To be honest, because I came of age in the era of LLMs, I will freely admit I’m still not sure I “know how to code”. Yes I can solve leetcode questions and when I put things into my IDE at work, what comes out of it leads to a production level service, but I still lean on coding tools and online resources a ton to help me figure out how to do my work. That’s software engineering is an occupation of continuous learning is both a blessing and a curse. You are always being challenged to figure out and solve novel problems so you never feel stale, but you also never feel completely confident that you’ve actually figured out how to do your job.
So for me, the answer to “when can I say I know how to code” just became if I’m able to ship stuff for my job then I must be doing something right.
Advice if you want it
Recall I started out as a Data Scientist, mostly focused on analytics. I worked for a really cool startup called Scoop that got decimated by the COVID pandemic and resulted in a mass layoff (around 80%) of the company, including the total dissolution of the Analytics team, but not the Engineering team.
It was then I realized that as an analyst, I could never be employee #1 at a startup, I could never start from zero. And that made me feel jealous of those people who could, and afraid of what I then perceived as my relative unimportance to the organization’s core function1. I very badly did not want to feel that way again and so I started this journey I’ve now told you about in two parts.
It was not easy and it was not linear. There were periods of stagnation where my job required me to keep doing the same kind of work that I did not feel like I was learning much from anymore, but I was persistent and continued to emphasize what my goals were to my managers over the years. Eventually that paid off as as persistence intersected opportunity and so when the org needed someone to make a technical transition, they found me.
The journey has also not been painless. Every time I went from Researcher to Analyst to ML Practitioner to Engineer I was gripped by the same fear and the same thought “maybe this is where I top out”, “maybe I’m just not cut out for this next challenge.” That fear would last a certain amount of time, I’d ship something or enough things to prove to myself I could do the new job (and prove to my boss I didn’t need to be fired), and slowly from there my confidence would improve.
Reassuringly these fear cycles also got shorter and shorter with each step I took: when I first started doing ML I worried for about a quarter but as an Engineer that doubt only held me for a little over a month. Also just remember that imposter syndrome is something everyone experiences, overcoming it is just part of the job.
Summing up
So no it was not easy but the view was worth the climb2 and I’d recommend it to anyone.
My journey is certainly not over, but I feel I’ve sufficiently broadened my skills that I can tackle enough problems without feeling out of my depth (and if I get anymore T-shaped at this point I’ll eventually just become a 2-D plane). Now I get to start going deeper and digging in on subjects I too hastily ran past during this 4+ year journey.
I come away feeling grateful for how well rounded I now feel having worn so many hats over time, and feeling grateful to everyone who gave me a shot to take the next step up the ladder; something I feel is becoming increasingly rare in tech as credentialing has become more important.
I am NOT saying analysts or analytics as a discipline is unimportant! I am just describing how I felt at the time I was laid off and the fear that motivated me to keep learning.
Aphorism attributed to John Stevenson.