It's harmful in that there exists a significant and vocal subset of users who does not wish to see that content or does not wish their children to do so. It's easier to teach your model never to produce that kind of content than to teach it to perfectly distinguish whether this user should see that content or not. TV channels are barred from broadcasting this kind of content for similar reasons.
Sure, there are always jailbreaks, but then the narrative changes from "we made a model that tells erotic stories to children" to "this ingenious teenager figured out a way to hack our model to make it produce erotic stories." In other words, jailbreak move the fault from the model producer to the model user.
It's also worth keeping in mind that erotica comprises a surprisingly large portion of fiction easily available on the internet for free, and "unfiltered" models tend to produce that kind of content unprompted (see e.g. the original Mistral). The major AI labs are probably filtering it out, but I suspect they can't go too far there, as having a model that is good at fiction is something they actually want.
Then there are the non-chat-gpt-app use cases (like customer support chatbots, automatic summarization etc), for which unprompted erotica is highly inappropriate. Those are the "business travelers" of AI, not the first thing one thinks of when talking about who uses AI models, but extremely important nonetheless.
I heard of this described as the minority effect, that a small minority can have a disproportionate impact. The example given is that it's cheaper to make all instances of a product kosher or halal than to make an entirely separate product.
All of this is true but then it's as easy as releasing censored and uncensored versions of the model.
Then it's up to users (or parents, in the case of children) to choose the adequate version for each purpose. Just like there are child-friendly movies and adult-only movies, and no one beyond fringe puritan crusaders would say that the latter should outright not exist.
Well here you still have the same problem, since they're not gonna release an actually uncensored version, that tells you how to do awful things (or indeed, that tells you to do them).
So then you'd have censored and less censored, and it would still be a matter of where to draw those lines.
True, "uncensored" is not the best term for what I meant (as I'm aware that fully uncensored is not a realistic thing to ask from companies).
What I mean is a model for all audiences and an adult model, and the line would be drawn at the law of the country producing it (if it's something that would be legal to publish for a human author at a website, then it should be allowed as an LLM response). So erotica would be fine, while instructions for making a bomb wouldn't.
Companies release uncensored models all the time. They're called "text" models. I just had llama3.2:3b-text-fp16 give me step by step instructions on how to make a pipe bomb.
I think it's easy to released the uncensored version, it's just the censored version that's likely super super hard.
Since this is just giving the model directly, there's no ability to do any filtering as part of inference, so I would imagine you have to assume the worst (intent) on any input coming into it.
There are also some practical constraints, like any kind of erotic content is completely prohibited in some regulations (like India), so if you want to be able to have access to human labeling or deploy the model under these regulations, you do need to comply.
It’ll get easier once the costs of building foundational models go down and human labeling gets automated. Sit tight, models that’d be creative and amazing at generating erotic content are certainly coming.
> It's harmful in that there exists a significant and vocal subset of users who does not wish to see that content or does not wish their children to do so.
"I have a right to live in a society that perfectly adheres to my personal morals" is not how companies or people should operate in a pluralistic society, despite Nassim Taleb's claim that the intolerant minority wins.[0]
>It's harmful in that there exists a significant and vocal subset of users who does not wish to see that content or does not wish their children to do so
It's hard to think of a scenario where there's a child technical enough to run Gemma 3 locally but somehow unable to access any other written erotica. Project Gutenberg is full of erotic textual content and I haven't heard of anyone calling for that to be banned.
>Then there are the non-chat-gpt-app use cases (like customer support chatbots, automatic summarization etc), for which unprompted erotica is highly inappropriate. Those are the "business travelers" of AI, not the first thing one thinks of when talking about who uses AI models, but extremely important nonetheless.
And how many of these are going to be using Gemma, when Gemini over the API is cheaper, faster and easier to use?
> It's hard to think of a scenario where there's a child technical enough to run Gemma 3 locally but somehow unable to access any other written erotica.
The reason you're struggling to understand is that you're thinking about this logically.
Adult content is obviously freely available to any child or adult with minimum technical skills. What makes LLMs different is that it's "the new thing" and people respond differently to "the new thing".
It's funny because the results are in, millennials grew up with pretty easy access to all manner of porn from an early age and the effect has been nothing. Even a reduction in intimacy if anything.
I'm sure the hysterical puritans of the past will come out any day now and admit that they weren't even 1% correct in their assertions.
It's what they switched when confronted with evidence, roll the clock back 10, 20, 30 years though and it was "Will turn them into rapists, molesters, and social degenerates."
No, there's no movement to shut down pornography on the internet. There's a movement to shut down specific websites and make a lot of noise about it but continue consuming pornography behind closed doors.
People like pornography. They'll as soon ban alcohol again (which worked so well last time)
Sure, there are always jailbreaks, but then the narrative changes from "we made a model that tells erotic stories to children" to "this ingenious teenager figured out a way to hack our model to make it produce erotic stories." In other words, jailbreak move the fault from the model producer to the model user.
It's also worth keeping in mind that erotica comprises a surprisingly large portion of fiction easily available on the internet for free, and "unfiltered" models tend to produce that kind of content unprompted (see e.g. the original Mistral). The major AI labs are probably filtering it out, but I suspect they can't go too far there, as having a model that is good at fiction is something they actually want.
Then there are the non-chat-gpt-app use cases (like customer support chatbots, automatic summarization etc), for which unprompted erotica is highly inappropriate. Those are the "business travelers" of AI, not the first thing one thinks of when talking about who uses AI models, but extremely important nonetheless.