the models only need to be right some of the time and they still constitute a threat.

Right!?!

And with how cheap and fast they are to run, they don’t have to be right that often as long as a separate run can verify/validate claims. In a lot of ways what the LLMs do should probably be thought of as just a very sophisticated sort of brute force approach. You’re effectively just searching the space of combinations of known vulnerabilities. It doesn’t require particular cleverness, just an encyclopedic knowledge, a reasonable way to prune the search space, and the ability to put in the compute time.

I must be missing something, because you’re getting a bunch of upvotes. I don’t disagree with anything you said, but it feels like a total non-sequitur to what I was saying.

Yes - proper air-gapping is a good idea. But asking the big AI labs to air gap is kind of pointless. Liability already gives them legal incentives to avoid incidents. If that didn’t stop them, a letter signed by a bunch of academics sure won’t. More importantly, open-weight models capable of doing all the things that we’re concerned about are already in the hands of bad actors around the world.

This letter is asking us to lock the front door of the house when the back wall is missing.

The ones in Russia definitely are. They don’t have to be local models - you run them in the cloud like anyone else.

The idea that the frontier labs have LLMs with special capabilities that make them dramatically better than the top open-weight models is largely marketing.

The reality is that most systems are poorly secured, and LLMs make exploiting vulnerabilities fast, cheap, and easy. What would have taken a trained security researcher weeks before can be done by a much less skilled attacker with an LLM in a few hours.

This is stupid. Open-weight models that can largely do what the OpenAI/Anthropic models did are already out in the wild.

Do these people not understand that you can’t put the cat back in the bag?

Memorizing passwords doesn’t scale well.

You’ve likely got dozens of accounts. You’re not going to memorize a password for every account. Re-using passwords is a serious security problem though. Poorly secured sites get their password databases leaked all the time. If you use the same password on lots of sites, it’s basically a sure thing that the hashes are out there on the darkweb already along with your email, and your password is compromised.

Also, typing your email password into an unsecured library computer is a terrible idea. It is ridiculously easy for someone to install a keylogger and gather emails and passwords. In the past, your main protection was that it wasn’t worth anyone’s time to break into the computers, install keyloggers, collect lists of passwords, then figure out how to use them to get money. But with LLMs that can all be automated. It has become ridiculously cheap, and can be done by someone with a bot farm in Russia.

A password manager largely fixes the first problem, but at that point you need a device with your password manager with you to log in anyway. And the password manager doesn’t fix the second problem at all.

I’m just not convinced that mandatory hardware attestation is actually a real problem we need to worry about.

Like, yes, people with high security jobs will have that for sure. And they should.

But large companies that want customers? They want to make it as easy as possible for their customers to connect. They’re not going to force people to have a hardware device which if damaged or lost will lose them a customer. That’s just not what they want.

And… honestly, you should probably buy 3. Losing your main key only to find out that your backup key broke a month ago and you haven’t been testing it is a real risk.

They’re actually not tied to a device. They can be synced across devices.

Bitwarden, 1Password, Dashlane, Google, Apple, and Microsoft will all do passkey syncing for you.

We don’t yet have a good Linux native passkey store. I believe the credentialsd project is working on a spec for that. Once that’s up and running I would expect to see some options for syncing without relying on a third party service (though, you can self-host bitwarden right now). It’s definitely not seamless on Linux yet though.

They really aren’t a trap. They’re a well designed standard. The tooling around them isn’t quite perfect yet, but they’re a huge improvement over passwords.

People upvote because they hate change.

It’ll be really interesting to see how this dynamic changes post-midterms.

Right now it’s pretty clear Trump is doing his best to just avoid having Iran in the news.

After the midterms, incentives change a lot. Trump will never be up for re-election, and I’m not sure at that point if he cares much about public opinion. In some ways, he’ll have a lot fewer constraints. But… he’ll probably lose congress. He likes to ignore congress though.

I don’t know how this is going to play out, but I think we’re in for a real shit-show after the midterms.

It doesn’t even have to be only for known devices. Once you’ve compromised one device, you can just phone home to have a full powered LLM do your work.

The only real use cases for this would be on closed networks with no internet access, or networks where the traffic is closely monitored. Like, yeah, if I’m trying to hack the CIA, maybe I consider something like this. For the vast majority of devices and networks, this is inefficient and unnecessary.

I mean, it is kind of cool. But vanishingly few organizations have tight enough security that this is worth doing.

I think people are right that the solution here isn’t to try a different project.

Instead of being confrontational about it, though, I’d recommend a different aproach.

evelyn: .lol 😜

you: I know! I know. The branding is pretty silly. But this really is the right tool for the task. Unesco and Wikimedia use it. And we can do our own branding for $50/month. 🤷

A lot can be fixed by just acknowledging people. They’re trying to evaluate something they don’t understand, and having a reaction. It’s not even unreasonable. The right thing to do is just acknowledge and move on.

Usually that’s all it takes. Don’t let yourself be embarrassed when you’re being reasonable. If they won’t move on, that’s the point where I start thinking about job applications somewhere else.

Doesn’t mean they’re wrong!

I’ll just upvote the both of you. We all need to be more aware of our echo chambers.

Yeah. Once you spend an afternoon debugging a failing cron job, the 5 minutes needed to write the boilerplate starts to feel pretty ok though.

I’ve never had to spend an afternoon debugging a failing systemd timer in the years since I made the switch.

It’s not. But the systemd timer has more robust logging and monitoring, and much more explicit and readable syntax.

Once you get used to systemd timers, it’s really hard to see why you wouldn’t use them if your system has systemd. Unless writing crontabs by hand makes you feel like a hacker I guess. I’m not even being totally sarcastic about that - there is a fun retro feel to cron.

I think most the serious investors know. But, in a bubble, stock value can keep going up long past the point that makes sense. If you think that’s going to happen, it can make sense to invest even if you think it’s a losing play in the long run.

This should really make the timing more urgent for Anthropic. I was legitimately surprised when they decided to delay the IPO.

Also there’s still a real chance that Anthropic has some real value proposition that I’m not understanding, though I’ve yet to find someone who can make a coherent argument for what that looks like.

Right? This is going to be an interesting story to follow.

On the one hand, they don’t want to IPO right now, probably because they don’t think investors are keen to buy at the price they’re asking.

On the other hand, I don’t really see things getting better for them as long as cheap open-weight models that are almost as good as Anthropic’s exist. As more people figure this out, I think it’s going to get worse not better. So I don’t think they can afford to wait long.

I think it’s important to call out that this isn’t an issue with AI being brilliant at hacking. It’s that these systems are basically totally unsecured, and the main protection is that no one with the technical skill is bothering to try to get access.

This isn’t to say this isn’t a serious problem. We’re moving from a world where you need a trained security expert to spend non-trivial time to get access to these systems to one where anyone with access to an un-restricted agentic LLM can do this trivially and quickly.

As a civilization, we need to start taking security seriously.

GPL (and copyleft generally) gets way more attention than it should. In the wild, MIT and similar permissive licenses are by far the most common. Linux is on GPL, but that may be a historical artifact as much as anything. I don’t know if Linus himself would choose it if he were making the choice today.

The wider FOSS community, in my experience, leans permissive rather than copyleft. The copyleft people are just loud.

EDIT: To be clear, when I say I don’t know what choice Linus would make today, I don’t mean that I’m skeptical he’d choose GPL. I literally just mean that I don’t know. He’s been quite clear that he’s not a principled believer in copyleft, but it does have real advantages for a project like the Linux kernel.

“The outputs of this opt-in vulnerability scanner will be fully model-generated, without human review or triage,” Anthropic explained. “This will enable faster and more frequent scanning, but means that it is possible reports will be incorrect or invalid.”

When I saw the headline, I was wondering about this specifically. This may make this service not super useful.

My experience with AI security reviews is that they’re fantastic at finding faults, but they always find a list of things to complain about. If there are no real/serious faults they’ll start finding things that kind of have the same shape as a security issue, but really aren’t if you dig into them. I’ve regularly had an LLM generate a list of 10-15 issues ranging in severity from “nits” to “critical” where none of them were actual issues.

Periodic reviews seem like they could get annoying really quickly, becoming more of a maintenance burden than a help.