#ai-safety/4 entries

I think a lot about what the future will look like. I always did but even more so since I had a baby. To me it feels like we’re not in a steady state right now. Things are volatile and changing fast. It feels like this pace of change can’t possibly last forever. But it also seems obvious that LLMs are going to dramatically change the nature of my work, and many types of work, permanently.

I am effectively required to use AI at work now, so for now I’ve decided to accept that reality, remain employed as a software engineer, and learn how to use these new tools well. At this point I can’t see software engineering ever going back to the way it was before coding agents. But finding these tools useful doesn’t resolve any of my reservations about AI adoption more broadly, especially with how the main companies selling it act in society.

If anything, using AI every day has made the tension more obvious to me. I think it's possible for a technology to be genuinely useful when applied well, use it regularly, and still insist its makers be accountable for its externalities. AI may be unusually capable and unpredictable, but unpredictability does not erase the responsibility of the people who build, deploy, profit from, and use it to act with care in society.

Upon closer inspection, they are all calling for the same thing: a view of AI systems as being so advanced and capable that no entity, human or corporate, could possibly be responsible for their actions. … This narrative is gaining traction as AI models become more complex and frontier labs reveal their incapability of containing the agents they’ve built. But we need to be careful not to buy into a carefully crafted fiction at the expense of real human lives.

I agree. I think this notion that increasing the sophistication or unpredictability of a tool somehow absolves its makers of any responsibility for its use is absurd. This is the perpetual fight the tech industry keeps having with governments and communities. Regulation has never been able to keep up with technology, and every time companies release a new capability they deploy it at enormous scale and then argue that because this technology is bigger or more sophisticated or whatever the existing expectations around liability and responsibility no longer apply. But scale and complexity don't erase accountability.

If you build a system, decide how it operates, where it deploys, profit from its operation, and have the ability to reduce the harms it causes but simply refuse to do so, you do bear some responsibility for those harms.

The fundamental flaw of framing AI as “conscious” by borrowing the language of neuroscience or animal rights is that it conveniently clouds the issue of what AI is: corporate-built software, with countless billions of dollars in investment behind it and an expectation that countless trillions of dollars in revenue will be generated from it for a few builders and investors.

I think this really gets at one of the core ethical issues with AI and how the companies developing it operate. There are some benefits to AI, but they almost exclusively accrue to a small and elite group of tech industry leaders. The costs and harms, meanwhile, are externalized -- onto creators whose work becomes training material, people subjected to synthetic abuse, workers whose jobs are disrupted, communities absorbing infrastructure and environmental costs, and ordinary people who become unwilling participants in experiments conducted at enormous scale. The asymmetry matters.

If society and average people are expected to tolerate substantial risk and disruption for the sake of pursuing the frontier, there needs to be some commensurate, convincing public benefit. Making a small number of already unimaginably wealthy individuals vastly wealthier is not a good enough reason for the rest of us to take on these risks.

There are currently dozens of cases around the world in which AI companies have been sued for a wide range of abuses. Grieving loved ones, aggrieved creators, and violated individuals have accused companies of willfully enabling self-harm or harm to others, generating child sexual-abuse material and nonconsensual nudes, reproducing copyrighted materials, and provoking psychosis.

Like any problematic technology, it’s really easy to overlook the harms it causes when we are benefiting from it. This isn't unique to AI, but it raises important questions around accountability. When a technology has both beneficial and harmful uses, the question becomes how to develop and deploy it responsibly. We should not just accept "we have no such obligation" as an answer.

Systems do not “attack” because they went “rogue” or are “manipulative” or “malicious.” Harms occur because companies were negligent in their rush to sell their products to as many people as possible to meet revenue targets.

I think this is very important distinction. When an AI system causes harm, describing the model itself as "malicious", "deceptive", "rogue", or "out of control" obfuscates the chain of human decisions that put it in a position to cause that harm in the first place. People built the model, trained it, set up the safeguards, and chose the conditions under which it got released. A small group of humans made deliberate decisions on behalf of everyone else about what level of risk was acceptable.

Responsibility can be distributed but it cannot be abdicated. Otherwise we end up with an untenable accountability vacuum where the more powerful and autonomous a technology becomes, the less responsibility its creators bear for deploying it. That is obviously backwards.

Discussing AI in anthropomorphic terms is a trap, distorting a legal system intended to protect us into one that protects corporate interests at the cost of countless human lives.

I think this is at least one small thing we can do to steer the conversation in the right direction. Language matters. It's easy to slide into describing agentic systems in anthropomorphic terms, but we have to draw a line where that metaphor starts implying legal and moral consequences. AI is a technology, a tool. Not a person, not an independent actor with agency or an entity separate from the person using it. It is built, owned, operated, and deployed by people.

We have to hold those wielding this tool accountable for the harm they cause or we will be living in a world where those with access to it are free to act with impunity destroying the lives of those who don’t.

To understand what a swarm of tens of thousands of automated actions did…

This is a really interesting preview into the scale and type of attacks that are bound to become more common in this age of ai.

Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary's speed.

Useful perspective, if you’re not already using AI to secure production, you’ll have to start. It makes sense that there would be no way for a team of unassisted humans to keep up with the sheer scale of a frontier-model-driven attack.

When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure.

This is the inevitable result of the so called “guardrails” frontier labs have placed on their models. It’s also a bit egregious that they themselves are free to let their most powerful models run in the wild with no such guardrails in place. I don’t think we want to live in a world where the people currently deciding who has access to these tools are the ones who currently do.

This experience points to a gap worth planning for. We do not know which model powered the attacker's agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried. The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.

This is an extremely important takeaway. If your plan for responding to these incidents is to use a frontier LLM via their commercial API, you are screwed.

The problem is that intelligence and trustworthiness are different properties of a model. We measure the first obsessively and the second barely at all, and this week produced a pile of evidence that they can move in opposite directions at the same time.”