Software engineering is table stakes

I'm going through a course on data engineering on Coursera right now and it's been super interesting. I've been one of those software engineers who works with data for a while now, and while I feel like I have a solid grasp of software engineering, I often feel like I'm fumbling my way through the more data-processing heavy parts of my job. I get that feeling like there must be a better way, but I don't know what it is, so I just put something together based on my paradigm as a software developer, and it often ends up being a sub-optimal solution.

There's an entirely new field emerging as a result of this proliferation software engineering roles that are mostly just about data processing, and it seems like the industry is settling on "data engineering" as a term for it. I've heard this type of work described in all kinds of other ways (data science, data analytics, ML engineering, ML ops, data ops). The interesting thing to me is that in all of these emerging disciplines, software engineering is just table stakes. It's assumed that to do any of these roles you can already read and write clean, maintainable, reliable, production-ready code. It's just interesting to me how the industry evolves.

When I first started writing code nearly a decade ago now, it was still not a given that everybody wrote tests, and the thought leaders of the industry were people like Sandi Metz, Katrina Owen, Sarah Mei, Martin Fowler, and many other other brilliant people writing and speaking about writing good code. Conferences were full of talks about good programming styles and patterns, and people were soaking all of it up.

It feels like there's this baseline assumption now that we've collectively absorbed all of that wisdom, distilled it to a few key best practices, and now it's literally the bare minimum expectation. It feels like software teams do mostly follow at least basic best practices now, but there is definitely still a long way to go. I'm curious to see how these roles evolve and what role software engineers will play as the need for more and more specialization continues.