Links/36 entries

I came across this paper through another Coursera course I'm exploring about reproducible research in preparation for a talk I'm giving in March and found it both enlightening and a little depressing. Enlightening because it offers an interesting perspective on some of the problems with the current state of data work that contribute to the replication crisis. As a software engineer, it's wild to me that it was not just acceptable but totally normal and inoffensive to publish exclusively the results of a computation, without providing any data or code that could allow someone else to reproduce the results. This is improving since the time this paper was published, but it is still very standard to find no indication of how a given analysis was run or result was achieved in a given paper.

The author describes one system their team at the journal Biostatistics implemented to tag papers with confirmed reproducible results which, at the time of this paper's publication, had been done successfully on 5 out of 125 papers. That was a little depressing to read. The author later points out that, like so many things in life, the standard we get is really just whatever the community is willing to expect and enforce.

I had a super interesting conversation with Ananth Packkildurai of Data Engineering Weekly today and a couple of interesting tools came up. One was this: dltHub. There are a lot of people trying to solve some of the main pain points in data engineering, and I'm convinced (like Ananth) that functional languages and approaches to software engineering have a lot to offer in solving them. This tool in particular appears to attempt to automate away some of the painful parts of the process, and I'm super curious to explore their approach. It seems like a really hard thing to do well. The problem of extracting data from traditional sources (DBs, files, APIs) and moving it elsewhere is well solved, but it's the increasingly complex transformations, large volumes, and disparate end use cases that are causing the strain with the current standard data stack.

One of my favourite blogs on the internet is Simon Willison's Weblog. I recently read this post of his about link blogging and love the idea. I occasionally post longer-form articles or essays on my blog, but usually only one or two per month, and I have a totally chaotic and unorganized web of ways to keep track of other interesting things I'm learning about or doing, but a lot of them are things I would love to share or connect with other people about.

I think his style of blogging is a great fit for the ways I use the internet and keep track of notes. I'm already working on refreshing my website (again), and I think this way of thinking about what to publish and how will really work for me. I'm still enjoying the experience of writing in Obsidian, and I find the built-in daily notes functionality makes it really easy to keep things organized. Now I just have to turn all the content into a website :) I'd like to also eventually migrate all of my various prior attempts at blogging, at least on technical topics, to this system so it's all in one place.

I know reddit is a real mixed bag, but I often find good information there. It seems like it's one of the last remaining places on the internet that's not dominated by SEO-optimized or AI-generated hooey. I started the Coursera course associated with the AWS Certified Data Engineer Associate exam and was looking for recommendations for more learning resources when I came across this. I haven't started it yet so I'm not sure how good it is, but in the past I've had a great time learning a lot from these types of community-made resources. I'm looking forward to checking it out.

It's also the first time I've ever heard of the DataTalksClub. I don't know anything about it but it's github profile makes it seem like type of online community I absolute love. I am super passionate about creating free learning resources for technologies I'm interested in and really admire people who have had success doing that in their own niches. I'm excited to join this community and see what happens!

This is a great book that appears to mirror much of the content from the Fundamentals of Data Engineering book by Joe Reis and Matt Housley. I'm finding it really satisfying this week digging more into the distinctions between data engineering, data science, data analysis, software engineering, and everything in between. I started out in very standard software development roles, building rails apps for various companies. My roles over the last few years have been decidedly not typical web development ones, but I'm never quite sure how exactly to explain what I do best. I feel like I'm finally on to something with all of this data engineering stuff.

I always wonder how useful these types of exams are in practice. It often seems like they're mostly not very applicable to "real" work, but this one (and the associated Coursera courses) seem well constructed and practical.

I started a new job a couple of months ago that's much more like what this course describes as "data engineering" than the type of software engineering I'm used to and I have become painfully aware of how little I know about the "cloud" and all the various hosted services for working with data. This course feels different than other ones I've come across so far in that it appears to focus on very hands-on and realistic types of work, and notably also tasks that are very similar to what I'm actually doing day to day now. I think it could actually be very useful to pursue.