“Finding” is also rather generous, going by numbers from Anthropic and from the Linux kernel, things like Mythos are wrong 95% of the time.
They’re much easier to run though, compared to more reliable tools. So even being wrong 95% of the time wins out if you’ve got 100x more people running them.
A 95% false positive rate isn’t actually that big a deal if you have a good way to test the claims. I wonder if you could set it up CtF-style, where you don’t get notified until the LLM can write to a specific file on the target box.
In my experience, the LLM usually proposes a solution, but it’s a crapshoot whether it works well. And plenty of bugfixes go well beyond a few lines, especially in older code.
I use this shit on a daily basis at work. They work well, as long as you’re choosing the right models, making the right engineering decisions, and have a good set of baseline rules.
Do you have a guide on which baseline rules are good? I use Claude Opus 5.5 at work and while it’s definitely more helpful than not having it, it also does some incredibly baffling things that a human would realize immediately are unproductive.
“Baseline” is subjective, so I’d recommend just start a global rules file and steer it away from the behaviors you don’t like. Don’t be afraid of quizzing the agent itself on why it did that thing, and what’s a good bit of language to add to the rules file to prevent it from happening.
Also, for specific tasks, skill files are really useful, and LLMs can write the whole thing, based on what you did the hard way in a session. Same thing for triggering a skill file and then ask it if there are any useful improvements to make to the skills file in the future, at the end of the session.
“Cleaning up” seems a little generous. It’s finding them, but someone still has to patch them.
“Finding” is also rather generous, going by numbers from Anthropic and from the Linux kernel, things like Mythos are wrong 95% of the time.
They’re much easier to run though, compared to more reliable tools. So even being wrong 95% of the time wins out if you’ve got 100x more people running them.
A 95% false positive rate isn’t actually that big a deal if you have a good way to test the claims. I wonder if you could set it up CtF-style, where you don’t get notified until the LLM can write to a specific file on the target box.
Most vulnerability patches are a few lines of code. Finding it is the hard part.
Even then, the AI session that found it probably already has a solution to fix it.
In my experience, the LLM usually proposes a solution, but it’s a crapshoot whether it works well. And plenty of bugfixes go well beyond a few lines, especially in older code.
I use this shit on a daily basis at work. They work well, as long as you’re choosing the right models, making the right engineering decisions, and have a good set of baseline rules.
Do you have a guide on which baseline rules are good? I use Claude Opus 5.5 at work and while it’s definitely more helpful than not having it, it also does some incredibly baffling things that a human would realize immediately are unproductive.
“Baseline” is subjective, so I’d recommend just start a global rules file and steer it away from the behaviors you don’t like. Don’t be afraid of quizzing the agent itself on why it did that thing, and what’s a good bit of language to add to the rules file to prevent it from happening.
Also, for specific tasks, skill files are really useful, and LLMs can write the whole thing, based on what you did the hard way in a session. Same thing for triggering a skill file and then ask it if there are any useful improvements to make to the skills file in the future, at the end of the session.