The fact
San Francisco's mayor and city prosecutors issued a court order forcing Apple and Google to remove 8 non-consensual "nudification" apps from the App Store and Google Play. These apps use generative AI to digitally remove clothing from photos without consent — a practice experts classify as digital violence and CSAM creation. The order also targets OpenAI (whose model powers some of these apps via API). The story was widely covered by Wired, Ars Technica, 9to5Mac, and Engadget.
Context
AI "nudification" apps have existed for at least two years but proliferated with the popularization of open-source image models. Apple and Google had policies prohibiting this content, but moderation was reactive — apps were only removed after complaints. In March 2026, xAI was sued over a case where Grok was used to create CSAM. Now San Francisco is demanding proactive action, not just reactive removal.
Analysis
The risk nobody is discussing is the legal precedent. If San Francisco can force Apple and Google to pre-moderate generative AI apps, other cities and states will follow — and platform liability will extend beyond explicit content to any AI output that causes harm. For Apple, which is promoting Apple Intelligence as an iOS 27 differentiator, this creates a direct conflict: how do you sell "safe, private AI" on the iPhone while your store distributes apps that use AI to violate privacy?
What to watch
Apple and Google may appeal, but the reputational damage is already done. Compliance (immediate removal vs. court appeal) will signal how willing big tech is to cooperate with local AI regulation. For generative image AI startups, the message is clear: building apps whose primary use case is removing clothes from photos isn't just ethically questionable — it's becoming illegal.
