Beyond Content Moderation: How One Trust & Safety Leader Thinks About Personal Risk on the Job
A Head of Trust & Safety at a mid-size technology company explains why protecting employees' personal information is part of protecting the platform itself.
Trust and safety teams do way more than take down bad content. For the people who actually do the work, that's only a fraction of the job, and often not the part that keeps them up at night.
This company's Head of Trust & Safety has spent more than a decade moving toward the problems other people avoid: account takeovers, fraud rings, angry users who show up at the office looking for someone to blame. Their path into the field started in security work at bars and restaurants in their college town, ran through a support and risk team at a major fintech company, and eventually landed in trust and safety leadership. It’s a role they've now held long enough to watch the field mature from something nobody had a name for into a recognized discipline with its own burnout statistics.
Neither the individual nor their employer are named in this case study for privacy purposes, though this individual is passionate about sharing his experience and expertise with the broader community.
Why personal risk became part of the job
What's the biggest misconception people have about trust and safety?
The biggest misconception is that it's just content moderation. Bringing down bad content is part of it, but I think about trust and safety in three buckets:
Keeping the platform safe from users
Keeping users safe on the platform
Keeping users safe from each other
Our job is making sure the platform is somewhere people can operate safely and within the law, without the product being strained or abused. That can mean disrupting a phishing campaign, flagging graphic content, or, in serious cases, reporting something illegal to authorities.
What led you to start thinking about your team's own digital exposure, rather than just the platform's?
Someone senior in the security organization had received a personal threat, and we were asked to look into tools that could help bring down information about a person that's out on the internet. That was the immediate trigger. But the underlying reason is that when you make moderation or account decisions, like suspending someone or shutting down an account, the person on the other end doesn't always agree. Sometimes those decisions make people very angry, and occasionally those threats become personal.
“I wanted to give my team a way to put a layer between them making a decision somebody doesn't agree with, and that person doing something harmful to them.”
Why Kanary stood out
What were you looking for when you evaluated tools?
I looked at a handful of options. I liked that Kanary was a woman-owned business, I liked the interface, and I specifically liked that it was configurable. I'm picky about having optionality and being in control of the technology I use, and security teams tend to be full of people who are picky in the same way.
How do you think about the split between what a tool can automate and what a person needs to decide for themselves?
Privacy isn't one-size-fits-all. Some people on the team like broad visibility online, and don’t worry about personal data on some sites or platforms. Others want much tighter control. I don't set a single policy for the whole team; I leave the decision to accept certain risks to the individual. What I worry about is the risks people don’t choose. For example, if someone is in a bad place emotionally and decides they want to find out where an employee lives because they're upset about an account decision, that's not a risk anyone signed up for. Some information is public record and there's not much that can be done about that. But managing exposures within our control is worthwhile to make it harder for bad actors to find and act on it.
Physical safety, reputation, and the limits of control
What kinds of harm are you most worried about?
I’m most worried about physical safety and reputational integrity, more than anything, tied to impersonation for financial gain. I'm not personally a high-value target for that, but I do think if someone wanted to damage the company, they could easily attribute a comment to me that I never said, and now the company is doing damage control for something that didn't happen.
The other risk is social engineering that uses personal familiarity as leverage. For example, someone impersonating a request from me to get a teammate to take down accounts, or wire money, because it looks like it came from someone they trust. It's rare, but attackers keep trying because they only need it to work once.
You've mentioned that you can’t fully control all risks. How does that shape how you think about this work?
If somebody wants to get into my house, they're getting in - I understand that. You put locks on the doors, but a determined person will always find a way. That's true of security generally; all you can really do is make the attack surface harder to work with. So the goal isn't eliminating risk, it's reducing how easy you make it for someone to execute by introducing as much friction as possible.
Making the case internally
What would you say to a security leader who thinks this kind of protection isn't their organization's problem?
The safety and security of your team is an organizational problem, full stop. So is a team’s mental health.
“Trust and safety is a tough job. It's one of the highest-burnout industries on the planet, and it has been consistently for decades. Giving people the ability to have more peace of mind so they can focus on their work, rather than worrying about their safety, is the move. For me, that will always be the move.”
It also matters for a less obvious reason: if people are quietly afraid of the personal consequences of doing their job well, that creates pressure to make softer, worse decisions just to avoid becoming a target. That’s an incentive structure you're building if you ignore it.
Advice for other organizations
What's your advice to other companies thinking about this for the first time?
Take the extra step to secure your accounts, even when it's annoying. Turn on two-factor authentication. Be deliberate about what you and your team post publicly. And think about the problem from the attacker's side: a lot of fraudsters aren't trying to get rich, they're trying to make ends meet, and they'll move on to an easier target if you make yourself annoying enough to deal with. You're not going to solve this for everyone. You're trying to be difficult enough that people go elsewhere.
What do you want more people in this field to understand?
That this isn't just about helping one company or one person get a little safer. It's about making the internet and the world a safer place, so the next generation has less of this to worry about. That's a much bigger and harder problem than any one company protecting its own brand, but it's the one actually worth solving.
If your organization is thinking about how to protect employees from doxxing, harassment, or targeted attacks tied to their work, learn more about how Kanary helps security and trust & safety teams reduce that exposure.
Book some time to talk with us.
