San Francisco's City Attorney's Office issued cease-and-desist letters to Apple and Google demanding the removal of 13 face-swap applications that use AI to create non-consensual intimate imagery. These apps are overwhelmingly used to target women and girls, creating serious harms including harassment, extortion, and reputation damage. The enforcement action treats these applications as illegal products that the platforms should not profit from or distribute.

This action represents one of the most direct regulatory moves against generative AI abuse in the United States, going beyond warnings to demand immediate removal and holding platforms accountable for distributing harmful tools.

What This Means for Your Business

If you operate a platform, app store, or marketplace, you need to urgently review your content moderation policies for AI-generated intimate imagery. Regulators are moving from guidance to enforcement. Establish clear terms of service prohibiting non-consensual deepfakes, implement detection capabilities, and prepare for potential liability if your platform continues to distribute these tools. This sets a precedent that platforms will be held responsible for AI abuse on their systems.