In March, a girl’s stepfather took his own life after cops discovered that he had used Grok to create 7,000 sexually explicit images using one photo taken when his stepdaughter was 11 years old, the amended complaint alleged.
Grok allowed the man to generate extreme images depicting incest and rape without flagging any harmful behavior, the complaint said. Seemingly, xAI’s child safety system only intervened after the man input a prompt for “gang rape.” That request sent a CyberTip to the National Center for Missing and Exploited Children (NCMEC), which alerted law enforcement to the AI CSAM.
Yet the harm was not stopped then, either. Despite mandatory reporting requirements to share information like a user’s IP address when CSAM is flagged, xAI repeatedly refused to help cops or NCMEC identify the user, the complaint alleged. For weeks, xAI allegedly “obstructed this investigation at every turn” and made it harder for “law enforcement efforts to locate, identify, and apprehend the perpetrator.”
Eventually, the stepfather was arrested after cops obtained a warrant to seize his devices. That’s when “a forensic review revealed approximately 7,000 AI-generated images and videos” depicting his stepdaughter, which were allegedly produced using Grok. Without Grok providing users with easy access to “undressing” capabilities, his family doubts he ever would have generated the harmful images, which he allegedly trafficked online in trade for “CSAM produced by other child sex predators.”
Musk should not be walking around free at this point.
Well, the only bright spot here is that the trash does sometimes take itself out.
All’s well that ends well.
xAI founder Elon Musk has denied that Grok has ever been used to generate child sex images.
Well, that’s a relief! Of course they wouldn’t do something like that. Because that would be super illegal!
… i hope he never had the chance or ability to actually do anything to her physically ._.
i hope learning about what someone did without her knowledge will not scar her for life…
How do you even generate ONE of those images and NOT immediately go to prison?
“FoR thE chIlDrEn” when privacy is on the line but they can’t put a weirdo in prison after generating SEVEN THOUSAN CSAM images?!
Hi, someone that has worked extensively in the trust and safety space before AI destroyed said trust and safety industry in the west here.
The short answer is the lack of human moderation.
The long answer is that due to companies using AI to detect CSAM (for some good reasons, but also because it’s way, way, way cheaper than a trust and safety team and their mandatory EAP benefits) it has become infinitely easier to get CSAM past filters these days.
This is especially true for tools that can generate CSAM from existing images, where both the generation and sharing of these images are likely run through the same detection AI, meaning if one gets through the other does as well.
GAN image detection models are literally a part of image generation training models; to explain better how a GAN model trains is:
Agent B is given [extensively tagged image of a scene].
Agent A uses random noise to generate [Random image of pixels]
Agent B then sees if [Random image of pixels] is [Extensively tagged image of a scene]
If not, then Agent A regenerates the image. Again. And Again. And Again.
Agent B only has extensively tagged images of scenes, and only knows what the end products should be… in theory. Agent A only learns what these tags mean and what these images should be based on feedback from Agent B.
So if you want to detect bad things… go with Agent B, right?
Wrong! Current GAN has AI-trained Agents in both roles, with manual additions to their databases to further refine their knowledge.
This means Agent A trains Agent B as much as Agent B trains Agent A. Some models even have an Agent C which uses a LLM to figure out how to describe an image based on the tags generated by agent A or B… which is then fed into Agent A or B to further refine the image.
Now which one do we pick for moderation? It doesn’t matter! It’s all an incestuous clusterfuck that is defeated by word play (given Grok is an LLM feeding into a GAN image generator, any tags the LLM gives the image generator will be automatically approved by whatever agent scans the image for CSAM, since the tags it can detect are already filtered out by the LLM).
Now you can see the problem with this. I can see the problem with this. Most human beings can see the problem with this. But this meets current global moderation regulations as long as there is a manual way to report content that gets past the AI moderation… which would work except without human moderators the AI moderation will still fuck up removal either by removing any reported content or just not removing content if the AI moderator disagrees.
Until regulators in every single market collectively decide to enforce human moderation teams on big tech or punish for content hosted regardless of the company’s efforts to prevent it, this will not change or stop. It will always be people exploiting the inherent arms race of GAN models to generate CSAM with minimal effort and no real way to prevent it.
Because it’s never ever about the children.
Totally not about the children, says robust discourse and literal thousands of images all about the children .
“Think of the children”
And this is what happens when they do that a little too much
it’s ironic how all the people barking “think of the children” turn out to be the people we absolutely DO NOT want thinking about children, ACTUALLY.
I think they think its the normal amount.
They are making their pedo dream land, with no laws or rules.
This is their so-called effective altruism.
We all know what to do.
We all know what to do.
So, if xAI did that, then did they get obstruction charges?
Why? This is just about a megapedophile, it’s not like anybody camped out on a lawn or asked Lockheed to stop selling f35s to baby murderers.
You would think the megapedophiles went extinct during the cretaceous era, not so!
Why would you think that? Do you live so far from an ocean?
You mean a slap on the wrist for the richest man in the world?
When you’re rich they let you do it.
Horrors of this aside… is Grok using a keyword filter for CSAM detection?
I cannot even begin to describe how technically negligent that is.
Image based CSAM detectors have been around for ~2 decades, and any image host with an ounce of self preservation uses them. I’m pretty use some Lemmy instances use them. Twitter must have already been using them, even before Musk bought them, because it would be legal suicidal not to.
That they wouldn’t implement a strong image-based checker in Grok is mind-boggling.
What about shareholder value tho?
Well, that’s what I’m saying. Even if shareholder value is the only objective, this is a catastrophic mistake. CSAM is baaad for business.
is Grok using a keyword filter for CSAM detection?
At some point in the last ten years, everyone apparently forgot why this doesn’t work
Not everyone. Not even AI platforms.
CivitAI is a great example; they have a robust image categorizer that auto tags anything uploaded to the site, or generated on it. They have it divided into a SFW and NSFW sites, and nothing even related to children is allowed on the NSFW side. No depictions of real people are allowed on either. And by all indications, they have CSAM firmly under control.
It’s why this story is so mind boggling to me. Either Grok’s team is such a mess that they weren’t aware of ancient filtering technology, or they deliberately left it “uncensored” and thought they could get away with it. Either possibility is horrifying.
A pedophile would remove a CSAM checker? Color me shocked.
He is such a…
A fucking…
LOSER.
This is what capitalism leads to.
Some fucking dweeb with more money than half the world combined.
Capitalism is INCOMPATIBLE with democracy. It’s not a system of resource distribution, it is a form of government.
I cannot even begin to describe how technically negligent that is.
Tbh, this is par for the course for most of Elon’s businesses
We need to do The Thing.
But even unintentionally serving CSAM is unique in that it can get you in legal hot water reeeal quick. It’s why web services are so strict about filtering.
I just can’t wrap my head around this. Maybe I’m misreading what Grok did, or maybe the story isn’t quite right, but if it is, the company should be getting raked over hot coals by the feds right about now.
Yeah, but remember who’s running the FBI right now.
O.O
More likely the Epstein class will be offered given “premium private” accounts.
xAI did more to protect a pedophile than it does to protect its regular users from law enforcement.
when the ‘leader’ of the country is a known csam perpetrator, i’m sure it emboldens those members of the public with similar mental leanings
Techbros in the past were infamous of having a “free speech absolutist” view, and saw CSAM as “recordings of crimes”. While this view fell out of mainstream (alongside with nonconsensual nudes being more ostracized), it wasn’t without a lot of opposition, and an almost irreparable harm to free speech advocacy.
here here … I concur
Yeah, this guy sort of digitally pimped his wife’s kid.
Which is a lot.
I’m thinking this has to be one of the most anti social behaviors.
However, there’s also, this Grok LLM. Which, tbh has said some shady shit on the internet and was abetting(?!).
That’s somehow more than a lot.
What a fucking headline. At least it had a happy ending.
Every image Grok generates of CSAM is another reason to rightfully call Elon a pedophile for knowingly allowing it to happen.
I really hope France fucks them up. In France, there is no question about the illegal status of fictional csam, it is illegal.
Those idiots think this should be legal, but not when the civilians do it. Tim Sweeny probably is only piping up because he insist on posting the stupidest thing his brain can come up with every day.
In no way shape or form has elon tried to dampen or stop this from happening to this day. He just made it so only paid users not in the united States can keep producing csam.
And trump wants this. This is the mysterious Ai progress we keep hearing about, next to price fixing, and survalience. States are not allowed to regulate Ai, and people will tell you that you are a dangerous person for trying to suggest that elon musk should be held accountable.
We are not all here looking for the guy who did it or confused on why this is happening. Elon musk has created a csam generator and doesnt have to stop it. Instead he just occasionally has to send one of his super fans to prison for using it the way he intended to. All to say hes doing something to stop this problem.
7k images and videos is a lot to go unnoticed. Its almost like this is allowed on their platform and encouraged. This isn’t a “we can’t be held accountable for what our users post” problem. This is a elon musk left a loaded gun out on a table, and he should be held liable for not putting it in a gun safe. Especially when every single other mainstream llm doesnt have this problem. Its negligence to the point of being willfully ignorant. You can’t tell me they can ban you for a single comment about Israel but all the sudden their moderation isn’t the greatest when 7k images and videos where being produced?
Well that is what happens when you train an AI on the likeness of Elon
