Links/2026×/#llms×/5 entries

To understand what a swarm of tens of thousands of automated actions did…

This is a really interesting preview into the scale and type of attacks that are bound to become more common in this age of ai.

Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary's speed.

Useful perspective, if you’re not already using AI to secure production, you’ll have to start. It makes sense that there would be no way for a team of unassisted humans to keep up with the sheer scale of a frontier-model-driven attack.

When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure.

This is the inevitable result of the so called “guardrails” frontier labs have placed on their models. It’s also a bit egregious that they themselves are free to let their most powerful models run in the wild with no such guardrails in place. I don’t think we want to live in a world where the people currently deciding who has access to these tools are the ones who currently do.

This experience points to a gap worth planning for. We do not know which model powered the attacker's agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried. The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.

This is an extremely important takeaway. If your plan for responding to these incidents is to use a frontier LLM via their commercial API, you are screwed.

This was a great read. I also believe this to be true and am equally concerned about the consequences.

The reality is thus: the people in charge either have no plan, or see no path forwards other than keeping their heads down.

This is interesting and unsurprising, it seems like it’s just human nature to fall in line, regardless of how absurd the groupthink is. I’ve definitely seen it in every project I’ve been a part of over the last 3 years or so since LLMs became mainstream. Suggesting that we are not, in fact, on the brink of a revolution and should just focus on building something that actually works is heresy.

Frequently the failure is not related to AI itself, but rather that companies are terminally bad at running software projects effectively, and as I have remarked previously, AI projects are subject to all the failure modes of normal projects plus you can get everything right and then still fail because of the method's novelty.

This is what people like me have been saying since the very beginning. The first DORA report on AI adoption came to the same conclusion — AI is just another tool. It will amplify whatever practices you already have in place, good or bad. Most companies have no idea how to ship software and AI doesn’t change that.

They cannot buy sensible software, hire competent talent, communicate honestly with executives about the state of projects, or undertake any sort of sensible initiative.

It’s a shame to hear from someone who has sat at many tables with the big wigs that it has come to this. Again, not really surprising, but it’s unclear what the path forward is.

The intersection of AI tooling and open source projects is interesting to me. I’ve also come around to the reality that LLM-powered coding agents like Claude code or codex are pretty clearly game changing when it comes to how software gets produced.

AI is a tool, just like other tools we use. And it's clearly a useful one.

It may not have been that "clearly" even just a year ago, but it's no longer in question today.

I also agree this wasn’t true even a year ago, but the last couple of generations of models have seen some sort of threshold crossed that we didn’t know was there and they are legitimately very useful in my day to day work now.

They also fundamentally change the nature of open source software development. Now that writing code by hand is virtually an obsolete skill, where does that leave maintainers? And what is the point of open source?

Sure, the social angle of working on open source is important and often a very motivating part of the project, but in the end that's a side benefit, not the point of the project.

This was part of it, and a big part of what I love about it. To me it’s really enjoyable working on interesting projects with a globally distributed team of people who also enjoy the work. Part of it was also performative, a way to demonstrate publicly that you could write decent software. I think that is mostly over.

In the kernel community we do open source because it results in better technology, not because of religious reasons.

Maybe that just leaves us with this — writing open source because it produces better results. This doesn’t strike me as universally true though. I guess it depends what you mean by “better”, but I can think of many examples of software where the proprietary version is much better than the OSS one.

I really love seeing more and more serious engineering being put into making LLMs actually work in production. They’re too cool a tool not to use or integrate into consumed software, but nobody actually wants more shitty chat bots. Making them work reliably to actually make software better requires structure and effort, and this is one cool way of thinking about it I hadn’t come across before.

I always appreciate these kinds of candid analyses. I’ve become convinced that all the various things people refer to as “AI” are indeed very useful, but they don’t obviate the need for actual engineering. At this point it’s been many years since LLMs became broadly available and the benefits remain largely hypothetical and in the future but the harms are very much real and present today.