AI for Trust & Safety
Trust and safety is the one place where a model's false positives cost as much as its misses. Everything here is built around that trade-off, and around how fast a team can respond when a new abuse pattern appears.
Blocklists miss context, models miss this week
Keyword blocklists flood moderators with false positives while sailing past contextual fraud that never uses a banned word. Machine-learning classifiers read context well but do not know the scam phrasing that appeared on Tuesday, and retraining is not a same-day operation. Platforms that pick one approach end up with the failure mode of whichever they chose. The second problem is operational rather than technical: even a good detector is useless if new intelligence reaches production on a release cycle. When a novel scam appears, the useful question is not how accurate the model is, it is how many hours until the platform can respond.
What AI actually does in trust & safety.
The four workloads that carry the value in this industry, in the order they usually pay off.
Hybrid detection: transformers plus live keyword intelligence
Multiple transformer models classify text simultaneously and their confidence scores combine through ensemble scoring, which reduces single-model bias. In parallel, a dynamic keyword engine scans the same text with compiled patterns, so deliberate obfuscation through character substitution, spacing and leetspeak still matches. One API call returns the score, the matched terms and the classification. This is the moderation and scam detection API we built and run.
Per-category scores and per-surface thresholds
A single allow or block verdict is too blunt to be trusted. Scam and phishing, spam, harassment, hate speech, threats, sexual content and profanity each get their own confidence score, and the platform sets thresholds per surface, because a marketplace listing, a dating-app opener and a support message carry very different risk. Anything in the grey band routes to human review rather than auto-blocking, which is what keeps false positives low enough for moderators to trust the system.
Zero-downtime rule deployment
Keyword intelligence is maintained independently of the application and applied to a live in-memory cache through webhook-triggered updates, thread-safely, with no restart and no interruption to production traffic. A trust-and-safety team can respond to a new pattern the day it appears rather than the next sprint.
Latency and cost engineering
Moderation sits in the request path, so it cannot be the thing that slows the product down. Asynchronous processing, parallel model execution, LRU model caching and CPU-optimised inference let the service scale horizontally as a stateless container rather than demanding a GPU per node, which also makes it cheap to self-host when user messages cannot leave your infrastructure.
Moderation decisions have to be explainable to the user
Under the EU Digital Services Act, online platforms operating in the EU must provide notice-and-action mechanisms and give affected users a statement of reasons when content is removed or restricted, including whether automated means were used in the decision. That obligation shapes the architecture directly: a system that returns a single opaque verdict cannot produce a statement of reasons, whereas one that returns per-category confidence scores and the matched evidence can. We build for that from the start, retaining the decision inputs and the rule version that produced each action so a moderation record can be reconstructed and reported on rather than reconstructed from memory.
The practice areas this work draws on.
Trust & Safety questions, answered.
Because the two failure modes are different. A classifier generalises to context it has seen patterns of, and misses the specific scam phrasing that emerged this week. A keyword engine catches the new phrasing the hour you learn it, and floods you with false positives on its own. Running both and combining the signals covers each other's blind spot, and lets you ship new intelligence without retraining anything.
Yes, and it is designed for that. Inference is CPU-optimised and the service runs as a stateless container, so it scales horizontally without a GPU per node. That is what makes self-hosting economically realistic rather than a compliance concession you pay dearly for.
Per-category scores and per-surface thresholds, plus a grey band that routes to review instead of auto-blocking. The point of a confidence score is to let you decide what to automate and what to escalate per surface, rather than applying one risk appetite to your whole product.
Keyword intelligence deploys through webhook-triggered updates to a live in-memory cache, with no service restart and no interruption to production traffic, so a new pattern can be live the same day. The model layer moves on its own slower cadence behind it.