Kira Howe

February 20252 entries

1 post/1 link

February 26, 2025

Making data useful

How do we turn messy data into profit?

Making data useful is difficult. The whole point of data engineering/science/analysis/etc is to turn all of the endless piles of information we collect into something that people find valuable. I've spent the better part of the last decade in my career as a software engineer building systems that do this, and I've noticed that it is remarkably difficult to do well. There are many points where the process breaks down, and no silver bullet for fixing them. I've spent a lot of time trying to organize my thoughts on the topic and wrote this down one night: […3 min read]

post#data-engineering#data-science

February 18, 2025

Data downtime (via)

I work with a lot of different kinds of data, and I'm very interested in the processes around how we transform the piles and piles of messy information that are so ubiquitous these days into useful data. I'm learning about data observability on Coursera and just came across this article that I think articulates many of the biggest problems in data engineering and data science right now really well. In particular this point:

Data downtime — periods of time when data is partial, erroneous, missing, or otherwise inaccurate — only multiplies as data systems become increasingly complex, supporting an endless ecosystem of sources and consumers.

This hits home. It's so easy to just pull a dataset out of anywhere now, but we rarely give any thought to whether the data in it make sense. Virtually every dataset I come across has duplicate and missing values, obviously incorrect values, and doesn't line up with its metadata. It's a huge problem, and beginning to untangle it is a very complicated problem but one I'm super passionate about.

link#data-engineering#data-quality#observability