Links/2025×/13 entries

This really hit home for me. Lately I've been feeling very overwhelmed and bored at work, feeling like I'm just drowning in AI slop and cannot possibly keep up. AI-generated PRs are flying through our previously robust process, production problems are way up, most of what I read day to day is not AI-generated, and it's frankly exhausting.

It feels like something has shifted just in the last few months. I'm not sure if it actually has, or if my organization has just taken up AI to a new extent recently, but somehow the day to day enjoyment I used to get out of programming has mostly disappeared. I've been reflecting on why that is and what I might need to change going forward, but I'm not sure what that is yet. For now I enjoy lamenting the old way of doing things with other old fogeys around the internet.

This was an interesting read with some useful advice on how to not go crazy at work. I've worked at lots of places and even the best of them come with their share of problems. It seems like there's just something about trying to get a large number of humans to coordinate on achieving a common goal that never goes very well.

The section on focusing on what you can control rings particularly true to me, especially this:

An easy way to burn out is to focus relentlessly on things you care about but cannot actually influence.

I realized a while ago that burnout isn't caused by overwork alone. Some of the most rewarding and fulfilling chapters of my life have also been the busiest. Burnout comes from the combination of overwork and lack of meaning. Working hard can be fine for a time if you feel like it's worthwhile. Working hard on something with no payoff or benefit and that you feel is pointless is what will slowly kill you.

This was a great read, an interesting takeaway is that we should be thinking about interacting with APIs differently when writing "agents" (I like Simon Willison's definition of "agent" as an LLM that runs "tools in a loop to achieve a goal").

Normally a human user of an API knows exactly what they want and makes a specific request for it. LLMs have no idea what they're looking for (they should have some means of validating or benchmarking the responses they get, but that's another too-often-ignored problem for another day), so they need to be able to iterate faster. Instead of sending a single (AI-generated) request, getting some (AI-generated) response, checking it, then repeating, we should be sending multiple requests and evaluating multiple responses in parallel before converging on the "best" answer. This means adding much faster and more robust support for branching queries and rollbacks, since in the world of agents these are extremely common compared to the rare cases they're needed during human interaction.

I work with a lot of different kinds of data, and I'm very interested in the processes around how we transform the piles and piles of messy information that are so ubiquitous these days into useful data. I'm learning about data observability on Coursera and just came across this article that I think articulates many of the biggest problems in data engineering and data science right now really well. In particular this point:

Data downtime — periods of time when data is partial, erroneous, missing, or otherwise inaccurate — only multiplies as data systems become increasingly complex, supporting an endless ecosystem of sources and consumers.

This hits home. It's so easy to just pull a dataset out of anywhere now, but we rarely give any thought to whether the data in it make sense. Virtually every dataset I come across has duplicate and missing values, obviously incorrect values, and doesn't line up with its metadata. It's a huge problem, and beginning to untangle it is a very complicated problem but one I'm super passionate about.

I love Randy Au's newsletter in general, but this one makes a particularly useful point: Not thinking about why you're doing something ahead of time is expensive and wasteful.

Randy's discussion of how this issue manifests in software teams in particular is completely on point, in my experience. But it applies to all areas of life. Software engineers get asked all the time to just "log everything", with the assumption being that since we have "big data" tools now, we'll be able to make use of it all later. The reality is that extracting value from heaps of unorganized, unstructured raw data is actually very difficult and potentially expensive, and the time to think about what questions you want to answer is before you start collecting data in the first place. It's also mostly true that once something is being collected or logged, it will go on forever. Another consequence of not thinking through your reporting and monitoring infrastructure ahead of time is that nobody really knows who is depending on what, and so things get left in place "just in case" it breaks something downstream. Of course this is madness, but it's how many organizations operate in reality.

It's true that storage is cheap now, but that doesn't mean accessing said storage is cheap, let alone doing something useful with the mess that's in there. Keeping terabytes of logs in S3 is like having a free warehouse in Labrador for all the tchotchkes you know you don't need but can't bring yourself to throw out.

A lot of this data hoarding stems from organizations wanting to be "data-driven", without ever explaining what that means for their context or why it matters. Collecting data is easy and cheap but does not inherently add value, and the difficulty of extracting value from raw data is a function of its volume (among other things). It's worth spending at least a little bit of time up front thinking about what

Rich Hickey just pushed a new namespace to the core.async library that looks really interesting. The accompanying rationale explains what it's all about in simpler terms than the code, but it feels like a big step forward in making core.async more intuitive to use. Channels are a super useful abstraction, but as the rationale explains, they are still a relatively low-level construct that require expertise and good architectural decisions to use well. This new namespace provides some more abstractions on top of channels that handle, among other things, "all channel I/O, thread lifecycle and coordination with the flow graph". This seems very powerful.

These are very high-level abstractions over an entire system, but I often wonder if this is just the inevitable direction in which software is headed. As we come to understand software systems better and better, and as the tools for generating implementations of solutions to known problems become better and better, it seems like this shift toward ever higher-level abstractions is the natural outcome.

It reminds me a bit of what Hyperfiddle is doing with electric Clojure, abstracting over the entire network, not just the "front" or "back" end of a web app. It's a totally different domain, but similar in that it feels like an impossibly high level to develop at, yet it appears to work very well. I'm excited to out both of these.

This is a really cool tool. I use LLMs extensively in my side projects with great success (my workplace is still investigating whether there are ways we can use them that will align with their privacy needs) and often simulate what this tool is purpose-built to do. I think it's going to save me even more time.

When I first started playing with AI-assistant editors (like Cursor, Windsurf, Co-pilot, etc.) I assumed they would be way better than chatting back-and-forth with an LLM, but it turns out, counterintuitively (to me anyway), that I much prefer chatting over the in-editor suggestions. I find these mostly intrusive and wrong, and they just create more work for me. In a chat interface, though, I can guide the conversation in constructive directions.

It might have something to do with the types of questions I ask them, or maybe just my personality. In any case, I frequently copy-paste dozens of files into a project, describe in great details a project's structure and data model, and then ask questions about it to an LLM. It sounds like this tool can automate and improve that process, which is great news for people like me who frequently engage in this kind of workflow.

I found this article interesting. It's about how AI-generated web UIs might help add more value to the work data scientists do. It was also a little depressing to read as a software engineer, but a good reminder of what our role is morphing into within organizations.

The main takeaway for me was that most code is so shoddy these days that we may as well just get LLMs to spit out UIs that are good enough to serve our immediate needs and then move on. She argues that between the break-neck hype cycle JS frameworks go through, high industry turnover, piles of existing tech debt, and constantly changing requirements, code written by LLMs isn't really that much worse than code written by developers anyway and given the astronomical cost savings, maybe we shouldn't be so skeptical about using generated, disposable UIs.

Of course my first reaction is cringe and outrage, but then I have to immediately admit that she's has a point. I've seen a lot of codebases, and I'm ashamed to say that as far I can tell the average quality of software in the wild is poor to terrible. The sad state of web development these days combined with the increasingly impressive quality of code that LLMs (especially Claude) can write, means I have to agree that it actually makes a lot of sense to use them to write software. They're at least very useful for prototyping and getting started, as she points out.

Ultimately this is just another reminder that software engineering is table stakes now. Anybody doing anything tech-adjacent in many organizations is more or less expected to know how to turn their output into a functioning web app a normal person can see, and despite our best efforts this is still an exceedingly complicated task. Nobody can be an expert in so many things at once, and as long as that remains the industry expectation, I predict that we'll continue to see more and more of the gaps filled by LLMs.

I came across this paper through another Coursera course I'm exploring about reproducible research in preparation for a talk I'm giving in March and found it both enlightening and a little depressing. Enlightening because it offers an interesting perspective on some of the problems with the current state of data work that contribute to the replication crisis. As a software engineer, it's wild to me that it was not just acceptable but totally normal and inoffensive to publish exclusively the results of a computation, without providing any data or code that could allow someone else to reproduce the results. This is improving since the time this paper was published, but it is still very standard to find no indication of how a given analysis was run or result was achieved in a given paper.

The author describes one system their team at the journal Biostatistics implemented to tag papers with confirmed reproducible results which, at the time of this paper's publication, had been done successfully on 5 out of 125 papers. That was a little depressing to read. The author later points out that, like so many things in life, the standard we get is really just whatever the community is willing to expect and enforce.

I had a super interesting conversation with Ananth Packkildurai of Data Engineering Weekly today and a couple of interesting tools came up. One was this: dltHub. There are a lot of people trying to solve some of the main pain points in data engineering, and I'm convinced (like Ananth) that functional languages and approaches to software engineering have a lot to offer in solving them. This tool in particular appears to attempt to automate away some of the painful parts of the process, and I'm super curious to explore their approach. It seems like a really hard thing to do well. The problem of extracting data from traditional sources (DBs, files, APIs) and moving it elsewhere is well solved, but it's the increasingly complex transformations, large volumes, and disparate end use cases that are causing the strain with the current standard data stack.