Can you elaborate on the last sentence a bit from a technical perspective? I’m not using code myself to the extent that I could really judge between LLM-generated code being good or just-works. I can just imagine compiler output being strictly deterministic, while LLM-code, due to the way a LLM works in principle, should be more …unstructured?
I honestly didn’t think you could even audit an LLM. It is possible, but not in the same way you would audit code. It is my understanding, because of the probabilistic nature of how the LLM operates, you can get different results even within the same model. This makes tracing something like logic next to impossible. Why the LLM gave the result is somewhat of a mystery. It is in effect a black box. This may change in the future, but for now LLM suffers from this issue.
You could probably trace how it comes up with the probabilities of the next token. Though the meaning of all the weights are probably going to be lost on you. Once you’ve got your probabilities and you’ve limited to some top number, it’s just a dice throw at that point.
Can you elaborate on the last sentence a bit from a technical perspective? I’m not using code myself to the extent that I could really judge between LLM-generated code being good or just-works. I can just imagine compiler output being strictly deterministic, while LLM-code, due to the way a LLM works in principle, should be more …unstructured?
I honestly didn’t think you could even audit an LLM. It is possible, but not in the same way you would audit code. It is my understanding, because of the probabilistic nature of how the LLM operates, you can get different results even within the same model. This makes tracing something like logic next to impossible. Why the LLM gave the result is somewhat of a mystery. It is in effect a black box. This may change in the future, but for now LLM suffers from this issue.
You could probably trace how it comes up with the probabilities of the next token. Though the meaning of all the weights are probably going to be lost on you. Once you’ve got your probabilities and you’ve limited to some top number, it’s just a dice throw at that point.
Sleeping agent in LLM is a scary topic and the tech itself has mathematically no solution for this: https://arxiv.org/html/2401.05566v2
Backdoors through model poisoning is so evil, I love it.