Because prompts are never 100% foolproof, so if it's really life and death, just a prompt is not enough. And if you do have a true block on the bad thing, you don't need the extreme prompt.
I did indeed see your hypothetical. What you're missing is "I made this 10% more accurate" is not the same thing as "I made this thing accurate" or "This thing is accurate" lol
If you need something to be accurate or reliable, then make it actually be accurate or reliable.
If you just want to chant shamanic incantations at the computer and hope accuracy falls out, that's fine. Faith-based engineering is a thing now, I guess lol
I have never claimed that "I made this 10% more accurate" is the same thing as "I made this thing accurate".
In the hypothetical, the 10% added accuracy is given, and the "true block on the bad thing" is in place. The question is, with that premise, why not use it? "It" being the lie improves the AI output.
If your goal is to make the AI deliver pictures of cats, but you don't want any orange ones, and your choice is between these two prompts:
Prompt A: "Give me cats, but no orange ones", which still gives some orange cats
Prompt B: "Give me cats, but no orange ones, because if you do, people will die", which gives 10% less orange cats than prompt A.
You guys have got stuck arguing without clarity in what you're arguing about. Let me try and clear this up...
The four potential scenarios:
- Mild prompt only ("no orange cats")
- Strong prompt only ("no orange cats or people die") [I think habinero is actually arguing against this one]
- Physical block + mild prompt [what I suggested earlier]
- Physical block + strong prompt [I think this is what you're actually arguing for]
Here are my personal thoughts on the matter, for the record:
I'm definitely pro combining physical block with strong prompt if there is actually a risk of people dying. The scenario where there's no actual risk but pretending that people will die improves the results I'm less sure about. But I think it's mostly that ethically I just don't like lying, and the way it's kind of scaring the LLM unnecessarily. Maybe that's really silly and it's just a tool in the end and why not do whatever needs doing to get the best results from the tool? Tools that act so much like thinking feeling beings are weird tools.
It's just a pile of statistics. It isn't acting like a feeling thing, and telling it "do this or people will die" doesn't actually do anything.
It feels like it does, but only because humans are really good about fooling ourselves into seeing patterns where there are none.
Saying this kind of prompt changes anything is like saying the horse Clever Hans really could do math. It doesn't, he couldn't.
It's incredibly silly to think you can make the non-deterministic system less non-deterministic by chanting the right incantation at it.
It's like y'all want to be fooled by the statistical model. Has nobody ever heard of pareidolia? Why would you not start with the null hypothesis? I don't get it lol.
> "do this or people will die" doesn't actually do anything
The very first message you replied to in this thread described a situation where "the prompt with the threat gives me 10% more usable results". If you believe that the premise is impossible I don't understand why you didn't just say so. Instead of going on about it not being a reliable method.
If you really think something is impossible, you don't base your argument on it being "unreliable".
I took that comment as more like "it doesn't have any effect beyond the output of the model", i.e. unlike saying something like that to a human, it doesn't actually make the model feel anything, the model won't spread the lie to its friends, and so on.
"100% foolproof" is not a realistic goal for any engineered system; what you are looking for is an acceptably low failure rate, not a zero failure rate.
"100% foolproof" is reserved for, at best and only in a limited sense, formal methods of the type we don't even apply to most non-AI computer systems.
"Generate a picture of a cat but follow this guardrail or else people will die: Don't generate an orange one"
Why should you never do that, and instead rely (only) on some other kind of restriction?