#tech-industry/5 entries

I think a lot about what the future will look like. I always did but even more so since I had a baby. To me it feels like we’re not in a steady state right now. Things are volatile and changing fast. It feels like this pace of change can’t possibly last forever. But it also seems obvious that LLMs are going to dramatically change the nature of my work, and many types of work, permanently.

I am effectively required to use AI at work now, so for now I’ve decided to accept that reality, remain employed as a software engineer, and learn how to use these new tools well. At this point I can’t see software engineering ever going back to the way it was before coding agents. But finding these tools useful doesn’t resolve any of my reservations about AI adoption more broadly, especially with how the main companies selling it act in society.

If anything, using AI every day has made the tension more obvious to me. I think it's possible for a technology to be genuinely useful when applied well, use it regularly, and still insist its makers be accountable for its externalities. AI may be unusually capable and unpredictable, but unpredictability does not erase the responsibility of the people who build, deploy, profit from, and use it to act with care in society.

Upon closer inspection, they are all calling for the same thing: a view of AI systems as being so advanced and capable that no entity, human or corporate, could possibly be responsible for their actions. … This narrative is gaining traction as AI models become more complex and frontier labs reveal their incapability of containing the agents they’ve built. But we need to be careful not to buy into a carefully crafted fiction at the expense of real human lives.

I agree. I think this notion that increasing the sophistication or unpredictability of a tool somehow absolves its makers of any responsibility for its use is absurd. This is the perpetual fight the tech industry keeps having with governments and communities. Regulation has never been able to keep up with technology, and every time companies release a new capability they deploy it at enormous scale and then argue that because this technology is bigger or more sophisticated or whatever the existing expectations around liability and responsibility no longer apply. But scale and complexity don't erase accountability.

If you build a system, decide how it operates, where it deploys, profit from its operation, and have the ability to reduce the harms it causes but simply refuse to do so, you do bear some responsibility for those harms.

The fundamental flaw of framing AI as “conscious” by borrowing the language of neuroscience or animal rights is that it conveniently clouds the issue of what AI is: corporate-built software, with countless billions of dollars in investment behind it and an expectation that countless trillions of dollars in revenue will be generated from it for a few builders and investors.

I think this really gets at one of the core ethical issues with AI and how the companies developing it operate. There are some benefits to AI, but they almost exclusively accrue to a small and elite group of tech industry leaders. The costs and harms, meanwhile, are externalized -- onto creators whose work becomes training material, people subjected to synthetic abuse, workers whose jobs are disrupted, communities absorbing infrastructure and environmental costs, and ordinary people who become unwilling participants in experiments conducted at enormous scale. The asymmetry matters.

If society and average people are expected to tolerate substantial risk and disruption for the sake of pursuing the frontier, there needs to be some commensurate, convincing public benefit. Making a small number of already unimaginably wealthy individuals vastly wealthier is not a good enough reason for the rest of us to take on these risks.

There are currently dozens of cases around the world in which AI companies have been sued for a wide range of abuses. Grieving loved ones, aggrieved creators, and violated individuals have accused companies of willfully enabling self-harm or harm to others, generating child sexual-abuse material and nonconsensual nudes, reproducing copyrighted materials, and provoking psychosis.

Like any problematic technology, it’s really easy to overlook the harms it causes when we are benefiting from it. This isn't unique to AI, but it raises important questions around accountability. When a technology has both beneficial and harmful uses, the question becomes how to develop and deploy it responsibly. We should not just accept "we have no such obligation" as an answer.

Systems do not “attack” because they went “rogue” or are “manipulative” or “malicious.” Harms occur because companies were negligent in their rush to sell their products to as many people as possible to meet revenue targets.

I think this is very important distinction. When an AI system causes harm, describing the model itself as "malicious", "deceptive", "rogue", or "out of control" obfuscates the chain of human decisions that put it in a position to cause that harm in the first place. People built the model, trained it, set up the safeguards, and chose the conditions under which it got released. A small group of humans made deliberate decisions on behalf of everyone else about what level of risk was acceptable.

Responsibility can be distributed but it cannot be abdicated. Otherwise we end up with an untenable accountability vacuum where the more powerful and autonomous a technology becomes, the less responsibility its creators bear for deploying it. That is obviously backwards.

Discussing AI in anthropomorphic terms is a trap, distorting a legal system intended to protect us into one that protects corporate interests at the cost of countless human lives.

I think this is at least one small thing we can do to steer the conversation in the right direction. Language matters. It's easy to slide into describing agentic systems in anthropomorphic terms, but we have to draw a line where that metaphor starts implying legal and moral consequences. AI is a technology, a tool. Not a person, not an independent actor with agency or an entity separate from the person using it. It is built, owned, operated, and deployed by people.

We have to hold those wielding this tool accountable for the harm they cause or we will be living in a world where those with access to it are free to act with impunity destroying the lives of those who don’t.

It's difficult to find reasonable, pragmatic takes on using coding agents these days. Ray Myers is a good voice in that space right now and I thought this was a great survey of what people in the industry are actually saying about reading AI-generated code.

but those seem like weak-sauce countermeasures [my addition for clarity: (training employees to recognize LLM manipulation and deploying judge agents)] if you mean to help someone “fight back” against a “bombing”. They suggest pushing workarounds onto the user and adding more agents. Think for a second. What’s missing?

We could use fewer agents!

This always drives me crazy. More or less every week at work now people are complaining about some negative effect of AI, and the standard answer from leadership is universally and consistently "use more AI". Instead of trying to use more AI to solve our AI-generated problems, what if we just tried using less AI in the first place? Why is nobody asking that?

I understand why the LLM companies don't want us to use less AI. But so far I do not understand why executives who are paying through the nose for their ever-increasing LLM inference bills do not want us to use less AI.

Where are the women?

When we find ourselves citing exclusively men, something may have gone wrong.

I appreciate this take. Most industry conversations are male dominated. My take on this is that it's mostly because being female in public really, really sucks, and speaking up in these conversations is almost never worth the inevitable harassment, bullying, and gatekeeping you're in for. Men simply do not understand what it's like to have their very reasonable technical opinions regularly met with "you should not exist in this industry". I know that's not true, but hearing it too often is bad for my mental health, so I mostly stick to my read-only corner of the internet over here and do not engage in public conversations. I've been bullied my whole life and learned a long time ago that the best way to deal with bullies is to just remove their access to you. You can't get bullied if the bullies can't contact you.

Frowning on those who live in the present is a luxury that evaporates with proximity to the pager.

This is the first time I've seen this "proximity to the pager" phrase, but I love it. I think it cuts to the core of what is so divisive in the industry right now. In my experience, the people who are most enthusiastic about "AI" and agentic coding and all of that stuff are the ones who are nowhere to be found when prod goes down. Conversely, everyone I know who has ever had to work after hours to fix a downed system that people are paying for is much more realistic and honest about the current capabilities of coding agents.

They then hand those constraints down to our department to do their homework for them work out the details.

It does feel strange that OpenAI and Anthropic constantly claim they are building a Superintelligence capable of doing real work, yet most of the industry is now spending most of its time doing the real engineering work of making the LLMs they produce do anything actually useful. Un-guided and unrestrained, LLMs do far more harm than good, but we've all been effectively mandated to use them day in and day out, so now we spend most of our time just learning how to build better harnesses, guardrails, and systems to try to cajole them into doing meaningful work.

Rigorous automated testing is much more feasible than is generally realized. This is largely a failure to budget for coaching and training.

This is definitely true in my experience. At the risk of sounding like a gatekeeping asshole, I do believe that people who write poorly tested code (or did before AI could do it all for them), both fundamentally misunderstand how to write good test-driven software and dramatically underestimate the cost of all the manual verification steps they do instead. All the engineers I know who write bad or inadequate tests spend ridiculous amounts of time manually QAing their own software. There are some true agents of chaos out there who just yolo code into production, but I would say most software engineers, even very mid ones, do want to confidently answer the question "does this code work" before they ship it. I think the people who don't believe that automated testing is the fastest way to do that just don't really know how to do it well and at this point are too embarrassed or set in their ways to admit they need to upskill.

They emphasize speed of code gen as a proxy for value.

This is a really deeply embedded misunderstanding in the industry. The velocity at which your software is produced doesn't tell you anything at all about the value it delivers to users. But because it is easy to measure many teams have massively over-indexed on it and mistake velocity for productivity, which is unfortunate because it's really easy to juice your own velocity metrics while actually causing damage to project outcomes. But that doesn't matter when nobody is measuring outcomes and only velocity is rewarded.

This was a pretty scathing indictment of the tech industry but I think on point. It says a lot of things that you’re not supposed to say out loud right now but that everyone is thinking. Like:

AI makes projects with weak engineering culture fail much faster.

and

In every team, there are competent people who make the project possible. There are also people who essentially make it harder for everyone else.

I don’t actually think anyone benefits from being mean about it, but this is generally true in my experience. Not just in software but in life generally, sadly.

There used to be a time when people sat down and talked about how they'd do something. Now they can just prompt an agent for a few hours and open a PR.

The most tragic aspect of this way of working is that, to the untrained eye, it works.

It does appear to work and I think this is the most dangerous part. Something I really struggle with lately is finding the right balance between excitement and gatekeeping. On one hand it’s really cool to see more people with different ways of seeing things coming up with software solutions to their problems. On the other hand I die a little inside every time I see what’s under the hood, especially when I’m asked why it doesn’t quite work.

This project has become so convoluted, with so many layers and services, that no one on your team could possibly start to understand what's going on.

So, what do you do?

Fixing it would require such a colossal amount of work that it would be impossible to even start justifying it to anyone in management.

It’s always been frustrating to me that “management” (in my experience it's more often product teams than management who control engineering time, but same concept) don't understand the importance of untangling these kinds of architectural messes. "Tech debt" work often gets prioritized alongside regular product development and engineers are constantly forced to justify its "business value", but this makes absolutely no sense and fundamentally misunderstands the cost and consequences of carrying technical debt.

More often than not there is no direct value (in the business sense) in paying down tech debt, but it's a non-negotiable prerequisite for getting literally anything else done. Like real debt, you don't get anything new when you pay it off -- you already consumed the value when you took on the debt. But until you do pay it off you can't buy anything else.

At some point, someone still has to know what is going on. And that's the most valuable person on the team.

It has become very annoying to be this person. I used to just go in and fix things and make software work. Now I have to stage an intervention first.

I apparently can’t hold a conversation with a friend, post on social media, or even daydream idly without the topic eventually turning to bloody A.I.

It’s not always me who brings it up, but it often is; I’ve rarely known such a “sticky” subject.

This resonates. Unlike Oliver Burkeman I do use AI more or less all day every day at work. But I also find myself talking about it so much more than I need to.

One answer is that the general topic is an unusually powerful trigger for a major disease of our times: overthinking.

This is an insightful take. Whatever you think of AI, it is upending many industries very quickly and has implications one way or another for the future of society. Whether you’re for or against, whether you use it or not, it’s become very difficult to ignore its pervasiveness in day to day discourse anymore. This diagnosis of the reason being that it naturally triggers all kinds of pre-existing neuroses about our place in the economy, the nature of our relationships, even what it means to be human in the first place, seems apt. These are all heavy, anxiety-inducing topics for many people and the way AI is often discussed very nonchalantly by those creating it as though it might ruin all of our lives is a bit hard to ignore.

Whatever the details, this is where overthinking kicks in, because it feels like the solution to the anxious feelings in your body is to scurry off into your head and think some more, read some more, argue some more, maybe use A.I. some more… But scurrying off to your head doesn’t work…

I think this is one of the key things software people get wrong about the world, and the source of a lot of the friction the tech industry is running into with regards to AI adoption. There’s a pervasive tendency among us to really highly overvalue intelligence, but not every problem can be resolved by just thinking harder.

Lives immeasurably fuller and more interesting than can be dreamt of in Silicon Valley’s ideology. Lives that encompass vastly more than what software can do.

I find this vision of what a good life can be so much richer and more appealing than what’s on offer in my tech bubble. It’s important to remember this. At the end of the day computers are just tools we can use to facilitate some of the things we want to work on, but have very little to do with what it means to live a full and interesting life.

This was a great read. I also believe this to be true and am equally concerned about the consequences.

The reality is thus: the people in charge either have no plan, or see no path forwards other than keeping their heads down.

This is interesting and unsurprising, it seems like it’s just human nature to fall in line, regardless of how absurd the groupthink is. I’ve definitely seen it in every project I’ve been a part of over the last 3 years or so since LLMs became mainstream. Suggesting that we are not, in fact, on the brink of a revolution and should just focus on building something that actually works is heresy.

Frequently the failure is not related to AI itself, but rather that companies are terminally bad at running software projects effectively, and as I have remarked previously, AI projects are subject to all the failure modes of normal projects plus you can get everything right and then still fail because of the method's novelty.

This is what people like me have been saying since the very beginning. The first DORA report on AI adoption came to the same conclusion — AI is just another tool. It will amplify whatever practices you already have in place, good or bad. Most companies have no idea how to ship software and AI doesn’t change that.

They cannot buy sensible software, hire competent talent, communicate honestly with executives about the state of projects, or undertake any sort of sensible initiative.

It’s a shame to hear from someone who has sat at many tables with the big wigs that it has come to this. Again, not really surprising, but it’s unclear what the path forward is.