LegitScript & Special Guest Marketplace Risk Panel Recap: How to Stay Ahead of Global Regulations and Trends: Using Deep Intelligence to Shape Policy and Mitigate Risk

Transcript

Luca: Well, let's go ahead and get started. 
Appreciate everyone's patience. It's always amazing coming off of listening to Erin speak. I'm still sort of galaxy brained thinking about everything she just covered and all of the updates on pig butchering scams, and the fact that she was on site is really amazing. But shifting gears a little bit. Welcome to Marketplace Risk. My name's Luca Jerone. I'm here with Legit script. We're proud to be a top sponsor of the organization and of this event. We provide deep intelligence and risk threat detection and certification solutions for partners like TikTok to help them stay ahead of changing regulatory trends and threats impacting their platform.

And I'm really excited for the conversation that we're about to have with Nuri Nusrat from TikTok and Ted James from Legit Script. I think a big reason why a lot of us get into trust and safety is because of the ability to make a difference in the work that we do. And I think we stay for the people that have passion and heart for what we're doing. 
So really excited to get into this conversation and talk about how you can think about a holistically approach to moderating content, to identifying risk, to incorporating regulatory knowledge and trying to stay ahead of those trends, and we'll get into some good conversation on why we do what we do as well. So with that, and apologies, I sprinted back here, so I'm a little out of breath. I'm going to turn it over to Ted to introduce himself first.

Ted: Hi, I'm Ted James. I'm an attorney with LegitScript, and I work in the research department, and I don't know, most of you know what we do. We're basically risk consultants. 
We help our clients assess their specific risks when it comes to regulatory risk, litigation risk, and
reputational risk. So we basically spend all day months monitoring platforms, monitoring merchants, monitoring online spaces, social media spaces, and whatnot, and at the same time, monitoring all sorts of regulatory develop developments around the world, just trying to track the emerging trends and where places are going and whatnot.

Nuri: And then, hi, everybody. I'm Nuri Nusrat. I work at TikTok on the U.S. Canada Regional High harm Policy Team on commercial activities, regulated goods and commercial activities, safety and civility and mental and behavioral health. But also, that's so that's what I do. 
But part of why I do this work is I was a teenage drug user from recreational to everyday user, and I'm very certain that had fentanyl been in the drug supply when I was growing up, I would not be alive today, so that's kind of my why of why this work matters to me, like, the drug supply is much more dangerous than when I was growing up. And when I was using drugs, we didn't care where we were getting them from. It was anywhere, right? 
Anyone and anywhere that's with something somebody that knew somebody. And it's just it's different now, right? People who are taking drugs for the first time or who are using it in an addictive way can all die from the fentanyl in the drug supply. 
And so that's part of my why, but I also, to break a little bit from the conference format, I know each of you in this room has a why about why you do the work. So I'm gonna offer up. It's optional, but if you are willing to, if you could find one other person, so a pair, it's going to be a pair share, if you could find someone sitting close to you, I'm gonna give each of you a minute to share why you do the work you do, and you share it to the limits of your own kind comfort and safety.

Luca: Thank you, Nuri, and appreciate everyone taking the time to do that. It grounds everything that we do, right, why we do it. So, thinking about the work that you do every day, Nuri, do you talk about just give us an overview of TikTok's approach to content, moderation, and the framework that you use in the different components of that?

Nuri: Yeah, and sorry, everybody. I'm a little nervous. I have my note cards. so I'm gonna read some off of the notes, but I'm trying to be human while I do it. So, as many platforms have, we have robust community guidelines that are the kind of the cornerstone of how we approach safety, right? And then we have tens of thousands of safety professionals, folks moderating in more than 70 different languages.

I think one of the things I love about TikTok that I didn't know before I entered tech was the experts we have run the gamut. So one of my closest colleagues is someone who's a former eating disorder therapist, and then we have folks from the DEA in law enforcement, former research psychologist. So there's a whole bunch of people that are kind of protecting the safety, but also thinking about what we want to allow on the platform and what we don't want to allow on the platform and in really thoughtful ways.

We also have specialized moderation teams for things like misinformation or other high harm things. So trade of drugs is something that we consider high harm. And we apply, as you can read on the transparency center, so sorry, some of this you might already know, but we use a combination of technology and human moderation, right? 
To take down content to identify content that might be violative of the platform standards, that might be ineligible for the four you feed or teen feeds, or that basically those two, I mean, those three categories. We also have global community partner channels where experts in the field can report to us content that they think violates our community guidelines. Then we have in-app reporting, which if any of you use the app, you can see kind of the process for doing that.

And then we also have something that I care about deeply. We have both a substance Support Safety Center, but also in-app kind of search interventions that redirect things like hashtag drugs to resources that we've created in collaboration with the crisis Text line and SA. So folks that are looking for a specifically kind of recovery related content, get redirects to those resources where they can actually see the numbers and text from your nap.
So that's, oh, sorry, that's the moderation. But also, we have moderation. It all depends on policy, right? So when it comes to policy, we're very strict as many of your platforms are about trade of drugs, right? 
Trade of drugs is always disallowed. But when I was thinking about things like, what do I want to stay here, I feel like it's very often that we focus on things like, what do we want to take down, but also what do we want to uplift, right?

And when it comes to drugs, kind of what I shared early in my life, the best of social media is that we want people to be able to connect with a community that's going through what they're going through at any stage of what they're going through. We want people to find education and resources and resources that can save their lives, right? So when we think about what we want to preserve on the platform, we want to preserve recovery stories. And that runs the gamut from people talking about where they're at and their recovery, what they might call relapsed or kind of back steps in their recovery. Like all of that is stuff that we want to preserve or counterspeech about drugs. What we want to take down is content trading drugs. So that's just kind of like the different buckets that the content falls into when we think about content moderation. Yeah. 
And we regularly update the community guidelines so that they are transparent.

Luca: Thanks so much, Nuri. Appreciate you talking about the different channels and ways that you both, you, prevent young people from accessing content that might be dangerous, but also directing folks to resources and recovery resources, et cetera, but making those connections, it's really, it's powerful. 
I think that aspect of things can get lost sometimes when we're talking about finding where the risk is. So appreciate you lifting that up. Ted, can you talk a little bit about how we, at legit script help platforms when we're thinking about regulatory and legal risk?

Ted: Sure. As I mentioned before, we help all sorts of platforms assess their regulatory litigation and reputational risk. And there's basically two parts of that for us. 
One is we're constantly monitoring products across scores of different platforms, you know, both monitoring the platforms specifically and also more globally, just trying to get a feel for the different products that are out there. We've got AI scrapers going all the time, and people much smarter than me within the company have built these amazing products for us where we can, watch these things go by in real time and watch the statistics and whatnot.

And when you know, I'm not sure how the AI stuff works works, but when it gets stymied on products that it can't recognize, we have human analysts that will take a look at them, and if they get stymied, they kick them up or down to us in the basement of the building at the research department, and we make assessments. So that's part of it.

Watching the products, watching for trends and products and whatnot. The other part is we pay a lot of attention to regulatory enforcement trends. Because what we're trying to do is we assess risk, we help our clients do that, we don't necessarily give advice, but just letting our clients know, here's where your risk is on your platform, you can decide what to do with it after that.

Sometimes you want to take a risk up, sometimes you want to take it down, but it's key that you know where your risks are. So anyone can tell you what the regulations are, you can Google that kind of stuff. We would like to watch the enforcement actions, because, you know, it's like it's driving your car. 
Anyone can tell you what the speed limit is, what you really want to know is at what point you get pulled over and get a ticket. Actually, my first job in a law school was prosecuting speeding tickets for a nearby city in the state of Oregon, it's 11 miles per hour, by the way. Maybe a lot of friends in that work too. So maybe that's not legal advice, but you really want to know at what point the risks kick in and what are the products that really bring a lot of risk and what are the products that aren't?

You know a certain product and it changes over time. You look at eyelash chemicals, theatopro analogues and whatnot. Technically, there's a prescription only. The FDA hasn't done anything about that since 2004, I think was the last time you got a warning letter on them. So, and they're ubiquitous. You see them on all platforms all over the place. But on the other hand, last year, there were a couple class action lawsuits about this. So you might not have regulatory risk, but you do have litigation risk and reputational risk on something like that is not existent.
Or you look at something like poppers, the muscle relaxants, like rush and whatnot, you know, by statute, those are illegal to sell to consumers for inhalation and purposes in the United States, similar in the UK, over in Europe. But again, you know, the FDA really hadn't done much since that one consumer warning, like six or seven years ago.

So a year ago, we would have told everyone, you know, there's some risk there, really, but they haven't done anything about it for four years. There's really no litigation risk and actually for
reputational risk, because these are fairly iconic in the LGBTQ community, you probably have more reputational risk if you pull these things down off your platform. That has changed this year under the new FDA, for a myriad of reasons, they've gotten very aggressive going after those popper products. They've raided several manufacturers, they've shut down several websites. 
So the enforcement has changed to the risk around those products.

So we spend a lot of time discussing changes in enforcement trends around the world for multiple different products and services, from drugs to financial services to get gambling to weapons. And then we sit around arguing about it all day long in the department.

Speaking of, not of the arguing, but of weapons and other emerging trends, I think we have a couple of examples of some recent things that we've been seeing. We'll get to it. Oh there we go. Yeah, so I said, there's a couple of ways we track trends. We're watching the products, you know, violative products are just products showing up on websites just for trends.
And then we're also watching enforcement actions. So there's these things. It's vaporizers, and rather than nicotine or cannabis, they have a narcotic used by hospitals for surgery, automodate, and automodate and automodate analogues. They're starting to show up in all these vaporizers. And we first started seeing them show up on various platforms we were monitoring, and at first it threw us for a loop,cause space oil, I mean, I'm not cool. I'm, you know, I'm the middle aged guy with kids. Like, I have no idea, like, what space oil is or capons. So it took us a while to figure out what was going on.

And then you started seeing enforcement actions come out of Southeast Asia, in Singapore of vaporizers general v very heavily regulated directionally against the law. 
And so when they catch people with them, they take them back to the police station and they test them. And then they put out warning letters. So countries like Singapore, Malayssia, and actually Australia are really good at testing vaporizer contents. So we get some good data out of that. We started seeing warning letters come out of them, and then the vaporizers started showing up in the U. UK and the Netherlands as well. So that's a trend that we're definitely keeping an eye on and I don't think it's really shown up in the U.S. too much yet. The reason they caught it in Australia, actually, is people thought they were smoking cannabis vaporizers, and then when they stopped, they'd have withdrawals. And that was the way the Canadian, the Australian regulators got involved.

It's the similar chemical that actually automodate is used by the state of Florida for their executions. It’s a little factor right there. And then the other thing we started seeing, we monitor a lot of social media, like chat rooms, as well as just to inform us of trends we're seeing, are these new pen guns. They call them bolt guns. They used to be zip guns. And it's like a pen and it's got a little bolt and you can stick a bullet in there and I guess you shoot someone with it. And we're starting to see those show up on a bunch of different platforms. We haven't seen any enforcement letters about it yet, because I think they're still kind of new, but it's definitely something we're monitoring, definitely something we're worried about.

We're seeing it more outside of the U.S., especially in Asia and Japan, but that's definitely a trend we're keeping an eye on. So that's basically what we do keep an eye on what we're seeing on the platforms, we're watching the enforcement actions, and, yeah, and that alerts us to what the scary stuff is out there.

Recently, we've also been seeing a bunch, not a bunch. 
It's very early, but we're starting to see PETN Pentophy, Pentophy. It's the explosive that was in the pager explosions over in the Middle East. It was used by the shoe bomber, it was used to blow up the Beirut Barracks way back when, and it's a plastic explosive, it's pretty dangerous. I guess it doesn't vaporize, so dogs can't smell it. And it, that's one of the reasons when you go to the airport travel internationally, they wand stuff to do the swabs and whatnot. And we're starting to see that show up on platforms. 
 And that's pretty scary. Most of the stuff, you know, the vapes and one that doesn't necessarily scare me, but when you start seeing that stuff, that shows up, it's, I think CAS number you were asking about CS number, CAS, 78115. Because that's how it sold a lot of times by the CAS number. 
But that's the trend we're definitely tracking and definitely concerned about. They said, yeah, and so with all of these different things changing all the time and pos popping up across the internet,

Luca: Nuri, how do you and your team stay ahead of this?

Nuri: So, I just listening to you... 
So I learned more every time I talk to Ted. This is only like the third time we've gotten to talk like this, but it's.. So there's brilliant people like LegitScript. So maybe I'll talk about what we do internally and then talk about actually what we do external ally. So we have internal, like many of your platforms, right? We have internal investigations teams that are always looking for new trends, new seed content, kind of creating new rules and detections to take down content.

But then there's folks that are working at LegitScript, working across platforms that, like Ted was saying, some of these things are emerging in different countries in different products, right? So they're able to alert us to trends before they might even make it to our platform. So we work with a robust kind of internal investigations, but for us, that's not enough.

We also want to work with external partners that have a kind of a landscape view of the whole industry, right? What's happening industry wide. And so I was some of the things that LegitScript gives us, we can use for our internal investigations rights, to do kind of deeper and more thorough investigations to add new seed content.

And so what I was gonna talk about with some of the regulatory support they give us, but I also just listened to you speak, I also, it feels important to say that you also help us, like, they help us assess the regulatory risk, the legal risk, but also the harm risk, right? Like when we're looking at content, what are the things that can take people's lives? What are the things that can make them experience addiction, right? 
What are the things that can kind of kind of like when you were talking with the eyelash, like that is still, we don't want it, but it is maybe less of our
priority when it comes to thinking about high harm, right? It's still a priority, but what are the things we want to focus on? So that is actually invaluable work.

So when I'm talking about regulations, it just feels really important to say that before I do it. So when they talk about something else they do help us do is keep abreast of kind of ever evolving regulatory landscapes. So, for example, with the FDA, right? 
When it comes to GLP1s, I'm sure many of you know, lots of our platforms saw an interest, like an increase in the usage and desire
for GLP1s, which we might know is like Maro, Ozempic, stuff like that, right? So there was a short one of the things Legit script supported us with was understanding the changing regulations.

So for a while, there was a shortage of the drugs from the manufacturer. 
So the FDA temporarily allowed those drugs to be compounded. The FDA, it's very rare that the FDA, and maybe you can correct me if this is what I understand. It's very rare that they allow drugs to be compounded because they don't test those drugs for efficacy or for safety, right? 
So they kind of gave like a certain, while these drugs are in shortage, compounding pharmacies can create these. And then, as some of those drugs are coming out of shortage, compounding is no longer allowed, but there's a wind down period allowed for compounding. So Legit script kind of gives us all of this information as it's happening in real time so that we can then take hard lines when it comes to moderation and keeping the platform safe.

So when they're telling us about the regulatory changes, we're able to take harder lines when people are talking about compounding pharmacies, for example, right? 
And just some of those drugs are no longer in shortage and some still are. So it's kind of ever rapidly changing, and we want to be able to kind of focus on our policy hardlines and trend detection so that while they supply us with the kind of information as is coming. I think. Yeah, I think that's all.

Luca: Thank you, Nuri. I think it is super important to recognize the landscape of risk in understanding, as you mentioned earlier, understanding what's happening across all of these different areas, which is one of the the key things that we try to provide to partners like you and have that feedback loop. As these trends are changing, the way that people selling these products are trying to promote them and the evasion tactics they're using to avoid detection.

Those are changing all the time, too. So Ted, what are some of the evasion tactics that you've been seeing lately and how have those tactics evolved?

Ted: Well, I guess there's two kinds of evasion that we see that we spend. I know, we spend time on. A lot of these products, you know, because the sellers want to communicate that they're selling the product. 
So they have to do that in some way. And in order to evade moderation, a lot of times they'll have, what's numbers written on pieces of paper in the background, or some of a lot of these products, like a lot of the GOP1 products, you'll see them sold in platforms for selling other products, like it'll be in the hardware section. Or it'll be, you on videos about a totally unrelated subject or they'll put them in the comment section.

We see a lot of that. We see a lot of videos where it'll be posted by the owner of the content creator, and then the content creator under other names, will be posting contact information in the comments. And it's a tough job for you moderators, because then you have to decide is it really the same person? Is this really just an innocuous video, but some random commenter is posting something about where to buy products. And that's where you have to do a lot of pattern recognition, you have to see, you know, it's the same person posting always on the same contact creators' creations, that sort of thing.

So we see a lot of evasion like that. The evasion that's almost more interesting for us is the evasion of people that feel like they found a legal loophole, and they're gonna that's the hill they're gonna die on. We see it a lot with GLP1s when they're sold for research purposes only or not a drug or not for human consumption. We see that a lot with steroids and bodybuilder drugs and bodybuilding drugs, and GLP-1s and some of the psychedelics as well.

The only thing about that is, you, the FDA and the DOJ, they were born at night, but it wasn't last night. And so they're not, you know, that doesn't fool them for a second. They actually have published rules about that, that it's a nonstarter, and in fact, there's been a few people prosecuted for selling bodybuilding drugs, claiming it for research purposes only. 
And because they claimed it was for research purposes only, they actually increased the penalty because it showed an intention to deceive and to try to evade law enforcement. So we see people trying to do that in the financial services space, we see a lot of people trying to know, if you're running a financial trading platform around the world, you' generally have to be registered in the jurisdiction where you're soliciting business.

We see a lot of people trying to use the exclusion of reverse solicitation, where they claim that these people just found them, they have no idea why they have so many clients in a certain jurisdiction, and they really don't need to be registered. 
And Luke has heard me rant about this forever, and if there's any financial services people here, you can listen to my rant going on after this meeting. But that exclusion is going away around the world. If you look at law enforcement trends, if you look at trends and guidance from different regulatory agencies or enforcement letters, there, that exclusion is going away.

There's another thing we see in the financial service spaces, there's new prop trading firms, you probably I don't know if you've seen them. It's the new trend where what they do is they claim they're a proprietary trading firm, but the definition doesn't mean anything in the Vulgar Act or
any country's definition of proprietary trading. And what they do is you give them $500 and they claim they're running an interview to hire you, or it's a trial period, and if you pass the interview and succeed, you'll have a job trading the firm's capital, although none of these firms appear to really have any capital.

But what they do is they take your $500, you can trade whatever you want. They'll give you 100,000 to one leverage. The moment you lose 1% on any given day, you lose all your money. The good news is you're welcome to apply again right away, and you can actually have multiple applications going at any one time. So that's one that's really thrown a lot of regulatory agencies and content moderators off. We've got some opinions from the SEC talking about the fact that this is a swamp product. 
And one of the we also, is that sell time? Yeah. I didn't know if you're doing the cut off.

Gambling is another one where we see I don't know if you've seen their sweepstakes, gambling is what it's called, and this one threw regulators off for a while, and it's where you buy tokens to play games on a website, which is fine. You buy tokens, you play games, you do the spinning wheel or the poker, but you can then take your winnings and convert them to something else fungible, either on that website or on another website. 
So you win the little silly tokens, but you can then go to another website and convert those back into cash. And you're seeing right now states around the U.S working on bills to ban that practice. So that's one that you know, they thought they'd found the magical loophole. 
You buy the coins, they're not paying you out in cash, themselves, it's someone else, who knows, but it's become prolific enough that numerous states that just saw Connecticut yesterday, filed the bill to ban them.

And Ryan Seacrest was just sued for promoting one of those websites. So that's another thing where they thought they found a loophole, and it throws content moderators off. And because these companies make so much money, they will send your company all sorts of legal documents and legal arguments about why what they think they're doing is not regulated, and it's really tough and it's kind of scary because they're making a lot of money, and they're willing to fight to the bitter end of what they're doing is legal. So that's the biggest evasion thing that we spend the most time on, probably.

Luca: Ted spilling all the tea on Ryan Seacrest? That probably wasn't on your bingo card for today, Nuri.

Nuri: I also was like, well, I never thought I'd be talking about Ryan Seacrest this morning. Cool. So I really believe in transparent community guidelines. I think a lot of platforms do really well at saying what's allowed, and my philosophy, I might be an idealist, is that most people, when I talk to creators, most people really want to follow the rules. 
They want to understand what they are so that they can follow the rules. So as I said, those are the cornerstone of our approach to safety. And those folks that don't want to follow the rules with transparent community guidelines, they will find ways that around them, right? 
So my philosophy is most people want to comply, and then the folks, when you are transparent, then that's when they try to evade. And that's that is a given to me. That's what's gonna happen.

And so I want to talk about false precursors. I know I keep bringing it back to fentanyl. It's really important to me, but I want to kind of right size, put it in scope, the problem in scope. 
So when it comes to the platform, TikTok, when it comes to alcohol, tobacco and drugs, 99.6% of content is removed before it's reported, and then 88.8% is removed before it's viewed. So it feels important to say the scope of the problem and to say when it comes to evasion tactics, when it comes to fentanyl precursors, there's always more work that can be done. And the evasion tactics, as many as anyone who's dealt with keywords knows that keyword evasion is always changing, right?

Like, you think you get ahead of it, and then there's zeros and O's and threes, and then they come up with something else, right? So the same thing is true of evasion tactics, right? So when it comes to fentanyl precursors, while I wanted to kind of scope the size of the problem, I want to say that the work is never done. It's deeply important to me personally. It's really, it's also deeply important to the platform, which is why I feel supported there, right? And so when it comes to evasion tactics or ventional precursors, part of our work with legit script was kind of understanding the landscape of precursors, right?

So the DEA scheduled the precursors. And like I said about community guidelines, when you know them, the folks that want to evade them evade them. So then when drugs are scheduled, people are like, oh, well, what if we can come up with some other non scheduled drugs? So there's about 150 fentanyl precursors. The material used to create fentanyl. So there's about 150 of those, but then people started creating precursors to the precursors. So, like, materials that you could mix with other materials that you could then make fentanyl, right, as an attempt to evade detection. And so some of the ways that shows up across platforms and on the platform, too, is like Ted was saying there's chemical abstract numbers, cast numbers associated with chemicals. So then folks, at first, maybe they were actually putting the whole chemical abstract number.

So those are some of the evasion tactics that are happening. And so what we use legit script for is both to kind of like I said, many times to give more seed content to our investigators. But what I find and to create trend rules to detect that kind of automatically through technology, but what is important to me as a policy person is I came to you and I said, I have lived experience with drugs, right? 
So I came into this job knowing something about what certain drugs look like, how I bought them right, on the street. And yet, and still, there's so much more for me to learn. That was also 20 years ago, so things have changed rapidly since then.

I remember coming in and having to kind of check myself, being like, oh, yeah, there are people in the world who don't know the difference between different types of drugs, so then it is my job to create training materials that keep them updated on like, what these drugs are, but what are the evolving evasion tactics?

So as LegitScript is able to kind of give us cross platform evasion tactics and tell us some before they reach our platform, then I am able to create materials that can train moderators to be like, "Hey, you know how like one month ago it might have looked like this? Now fentanyl precursors look like this. 
Now these are the evasion tactics, so we're able to kind of keep ongoing learning. And so when it comes to like proactive moderation, that content being taken down before it's seen or before it's reported, that's kind of where I really get to focus on as a policy person, and that's where both the content they give us in terms of visual signals, audio signals, textual signals, evasion tactics, like all of these are things that I can put into resources for moderators to make their jobs easier.

And for me to kind of undo my bias that people might have more experience or more know-how than, you know, why would they? Like, this is constantly changing, and so I feel really excited about that part of my job.

Luca: Thank you so much, Nuri. I think I'm just realizing we're at time, but I think that's a great place to wrap it up. 
I really appreciate you both sharing your expertise and knowledge. If you're curious about how we help TikTok and other folks in the space with getting an understanding of what risk is out there through risk landscape reports and our intelligence services. Come talk to us at our booth and with Nri's permission, invite you and encourage you to have a chat with her and ask her some questions. Ted, as well, clearly they know so much and they're in and out of this work every single day. So thank you so much, I really appreciate you both.