Is your app as efficient as what an experienced developer would create? If you released the source code, would it have security vulnerabilities? These are just a couple of the more hidden issues that fly under the radar when shipping LLM-generated code.
Is your app as efficient as what an experienced developer would create?
One of the earliest uses we had for LLMs was literally just asking it to optimize several large codebases. Lots of pointless changes suggested; several huge performance wins we had overlooked.
And all done – implemented, tested, and human-reviewed – in about a person-week, compared to at least half a dozen person-months to go through all that by hand.
I mean, sometimes the LLMs generate slow algos. But less often than human coders.
If you released the source code, would it have security vulnerabilities?
You’re not gonna believe this, but another of the first things we did was ask the LLMs to review the codebase for security issues (and review any new PRs)
OFC the code also gets reviewed for security vulns like it always has, by old-school automation (eg valgrind, fortify, yadda), human review, and red-teaming exercises. I don’t think I’ve seen enough data yet to say whether it’s got more/worse security issues than human-generated code (which, need I remind you, is often highly insecure)
These are just a couple of the more hidden issues that fly under the radar when shipping LLM-generated code.
Ummm… those would be issues if you didn’t use good orchestration, didn’t have good tools and docs for the LLMs to use, didn’t have follow good software engineering practices to begin with…
Is your app as efficient as what an experienced developer would create? If you released the source code, would it have security vulnerabilities? These are just a couple of the more hidden issues that fly under the radar when shipping LLM-generated code.
One of the earliest uses we had for LLMs was literally just asking it to optimize several large codebases. Lots of pointless changes suggested; several huge performance wins we had overlooked.
And all done – implemented, tested, and human-reviewed – in about a person-week, compared to at least half a dozen person-months to go through all that by hand.
I mean, sometimes the LLMs generate slow algos. But less often than human coders.
You’re not gonna believe this, but another of the first things we did was ask the LLMs to review the codebase for security issues (and review any new PRs)
OFC the code also gets reviewed for security vulns like it always has, by old-school automation (eg valgrind, fortify, yadda), human review, and red-teaming exercises. I don’t think I’ve seen enough data yet to say whether it’s got more/worse security issues than human-generated code (which, need I remind you, is often highly insecure)