Fraud by Prompt: Key Takeaways on Problematic Uses of AI at MRNYC

User-facing Problematic AI Is Growing Exponentially

Some of the most common violative AI is what users see on platforms. This ranges from adult and NCII content to fraudulent documents and copyright-infringing content. This is what trust and safety teams work at daily to keep off their platforms. There are also more gray-area uses of the technology, such as using AI to alter the photo of a product in a way that can mislead buyers. The challenge here is to set a good policy that allows teams to draw a distinction between innocuous AI usage and that which misleads others.

AI and Sexual Content Moderation

One of the most pervasive challenges in AI monitoring today is its use in creating adult content and non-consensual intimate imagery (NCII). Some of the issues trust and safety teams must face include:

• The subjective nature of sexual content. This makes it difficult to create broad policies that are easily applicable in every situation.
• Jurisdictional variation. Some states have removal laws around NCII and teams need to stay abreast of regulatory changes to be compliant.
• Victim outreach. Some laws require platforms to inform victims of NCII, but without contact information, it can be challenging to comply. 
• Lack of industry standards. There is no overarching policy for how platforms approach AI-generated content, especially around watermarking or provenance of generated content.

AI Evasion Tactics

The panel talked about how users, sellers, and developers are employing evasion tactics, both in hiding violative AI content and in using AI itself to mask other violative activity. Here are some of the techniques panelists noted:

• Cloaking. Website operators and developers in app stores are using cloaking, a deceptive digital practice where a website, app, or listing shows one version of content to automated bots or security scanners and a totally different version to human users.
• Adaptive design. Similar to cloaking, this technique involves content based on the type of device being used. Some bad actors also use geotargeting to avoid showing violative content in areas where enforcement is strongest.
• Gated content. Another technique is to show innocuous content on publicly accessible pages or listings and then reveal violative content hidden behind subscriptions.
• Scale. Bad actors are using AI to more quickly spin up fraudulent or violative websites, product lists, apps, and more to scam users.

Solutions for Better AI Monitoring

Panelists discussed how better management of problematic AI was something that requires both internal team efforts and industry-wide collaboration:

• Red-teaming. While typically a security exercise where a team acts as an attacker to test how well an organization detects and responds to real-world threats, it can also be a useful approach for violative content. By taking the role of bad actors, teams can try to imagine all of the ways users may try to circumvent policies and evade detection.
• Operationalizing policy. Because AI offers so many gray areas that test judgment, teams can benefit from clear, thoughtful policy that offers a roadmap for action regardless of the content that is flagged.
• Cross-platform collaboration. If platforms could align better on best practices around AI, they could create industry standards that users would become familiar with regardless of the platforms they are on.

Grow More Confidently With LegitScript

LegitScript's Platform Risk Solutions were built for nuanced and complex issues such as violative uses of AI. Our Ad Monitoring, Marketplace Monitoring, and Risk and Policy Advisory services give platforms the data, context, and expertise to catch what manual review misses, whether that's a cloaked listing, an AI-altered product photo, or a policy that hasn't caught up to the technology yet.

You don't have to choose between moving fast and staying safe. Talk to our team about how LegitScript can help you build a platform that's ready for what AI throws at it next.