User-generated content is created on your platform - text, images, video, audio, or live streams. This is the trigger point where the moderation journey begins.
- Text - messages, comments, reviews, bios
- Images - uploads, profile photos, attachments
- Video - clips, recordings, uploads
- Audio - voice messages, recordings
Sent to Checkstep via APIYour platform calls Checkstep's API with the content payload - the technical handoff where Checkstep takes ownership of the moderation decision. A small amount of engineering work is all that's needed.
- Content payload - the actual text, image, or media
- User ID - who created the content
- Platform identifier - which platform sent it
- Content type tags - so the right models trigger
Routed to your AI modelsCheckstep routes content to your configured AI models, which run simultaneously. Each analyzes content for its harm type and returns a label with a confidence score. Models only run on eligible content - text models skip images, and vice versa. Most platforms need 4–5 models.
- Pre-trained - e.g. AWS Rekognition for images
- LLM-powered - nuanced text classification
- Example-driven - trained on your content
- Keyword-based - exact term matching
Labels + scores passed to rulesEach rule defines: if this label comes back above this threshold, take this action. A single piece of content can be evaluated against multiple policies at once, and Checkstep routes based on the highest severity.
- Label - e.g. a "Hate speech" result from the LLM
- Threshold - e.g. confidence above 85%
- Action - e.g. send to the human review queue
- Policy - which policy this rule belongs to
Threshold determines the pathThresholds you configure decide the path. The 95% / 50–95% / under-50% split is a starting point, not a fixed rule - a child-safety policy might auto-enforce at 60%, while a spam policy only flags above 80%.
> 95%Auto-enforceHigh confidence - removed automatically and logged.50–95%Human reviewFlagged and routed to the Moderation Dashboard.< 50%Trust & allowLow confidence - published normally.Flagged content enters the queueModerators see the flagged content and flag reason, full conversation context, user history, metadata, and the specific rule that triggered. Sensitive media is automatically blurred and greyscaled for wellbeing, and ModBot can auto-decide clear-cut cases with documented reasoning.
- Enforce - remove the content
- Dismiss - no action needed
- Escalate - send to a senior moderator
- Suspend or terminate the user
A decision is madeEach action - automated or human - is recorded chronologically as a full audit trail: what was flagged, which model triggered it, at what confidence, and who acted. Checkstep then generates the user notification automatically.
- Audit trail - a full, regulator-ready history
- User notice - what was removed, which policy, and how to appeal
The user can appealThe user submits an appeal, the case reopens with the original decision visible, a moderator reviews it alongside the user's notes, and the final decision is sent automatically. The Transparency Portal is generated and maintained automatically - required under the DSA and Online Safety Act.
- User submits an appeal via the Transparency Portal
- Case reopens with the original decision visible
- Moderator reviews and decides: uphold or overturn
- User is notified of the outcome automatically
Everything feeds reportingDashboards track incident volume, breakdown by policy, auto-enforcement vs. review rates, and appeal outcomes - plus moderator performance and QA sampling that routes a percentage of decisions to secondary review to surface inconsistencies.
- Volume and policy breakdown over time
- Auto-enforce vs. sent-for-review rates
- Appeal outcomes - upheld vs. overturned
- Moderator accuracy, decision time, and consistency
Platform
Content journey
platform