Text Moderation & Scam Detection API

The challenge

Online platforms process millions of messages, listings, reviews and comments a day, and the harmful ones move faster than any static filter can track. Keyword blocklists alone flood trust-and-safety teams with false positives while missing contextual fraud; AI models alone overlook the scam phrasing and slang that appeared this week. Content moderation has to happen in real time without slowing the product down, and new intelligence has to reach production instantly rather than waiting for the next release. The objective was one production-ready service that handles both jobs — text moderation and scam detection — by fusing machine learning with continuously updated business rules.

What we built

A hybrid detection platform that pairs transformer-based language models with a real-time keyword intelligence engine. Text arrives through a high-performance REST API; multiple transformer models classify it simultaneously and their softmax confidence scores are combined with ensemble scoring, which produces more reliable predictions and reduces single-model bias. In parallel, a dynamic keyword engine scans the same text with an optimized in-memory pattern matcher built on compiled regular expressions, so deliberate obfuscation — character substitution, spacing, leetspeak — still matches. Both signals are evaluated together, and a single API call returns the confidence score, the matched keywords and the classification.

One API, two jobs

The same pipeline powers content moderation and fraud detection, because both are text classification problems with different label sets. It flags scam and phishing attempts, spam and solicitation, abusive language and harassment, hate speech, threats, sexual content and profanity, and returns a per-category confidence score rather than a single blunt allow/block. Platforms set their own thresholds per surface — a marketplace listing, a dating-app opener, a product review and a support message each carry different risk — and route anything in the grey band to a human review queue instead of auto-blocking it. That is what keeps false positives low enough for moderators to trust the system.

Built for production

Keyword intelligence is maintained independently of the application, so new fraud patterns and policy terms deploy instantly through webhook-triggered updates without restarting services or interrupting production traffic — rule updates are thread-safe and applied to a live in-memory cache. Low latency comes from asynchronous request processing, parallel model execution, LRU model caching and CPU-optimized inference, so the service scales horizontally as a stateless Docker microservice rather than demanding a GPU per node. That also makes it cheap to self-host, which matters when user messages cannot leave your infrastructure.

Why it matters

Trust and safety cannot rely solely on static blocklists or on a standalone AI model. Effective protection needs continuously evolving intelligence combined with contextual language understanding. Combining transformer-based machine learning with dynamic keyword intelligence gives platforms one moderation API that catches both known and emerging patterns, at the speed and reliability modern products require — and lets a trust-and-safety team respond to a new scam or abuse pattern the day it appears, not the next sprint.

Results