Documentation - Reference
Measuring what matters
A crawl, walk, run framework for Trust & Safety success - know which metrics to track at each stage of your moderation maturity.
The teams that improve fastest aren't the ones tracking the most metrics - they're the ones tracking the right metrics for their current stage.
Before you begin
Metrics only tell you something useful if you know what you're trying to achieve. This guide gives you a practical framework for measuring Trust & Safety performance at each stage of maturity - from your first week live through to a fully optimised operation.
You don't need to track everything at once. Start with the basics, build confidence in your data, and add sophistication as your operation grows. Each stage builds on the previous one - don't skip ahead until the current stage's metrics are stable and understood.
All of the metrics in this guide are available in your Reporting Dashboard. If you're not sure where to find a specific number, ask your account manager or check the Reporting Dashboard walkthrough.
Crawl
You just went live. Your job right now is to watch, learn, and understand what normal looks like on your platform. Don't try to optimise anything in your first two weeks - just observe and record.
Daily incident volume
How many pieces of content are being flagged per day? This is your baseline. You need to know what "normal" looks like before you can spot anomalies. Track the trend line, not individual days.
Auto-enforcement rate
What percentage of flagged content is being automatically enforced versus sent for human review? If this is extremely high, your thresholds may be too aggressive. If it's near zero, your thresholds may be too conservative.
Human review queue size
How many items are waiting for moderator review at any given time? If the queue grows faster than your team can process it, your thresholds are sending too much to review - or you need more moderators.
Average review time
How long does it take a moderator to handle a single case? This tells you about the complexity of what's landing in the queue and whether your moderators have enough context to make fast decisions.
False positive rate (spot check)
Of the content your team reviews and dismisses (no violation), how much is there? A high dismiss rate suggests your AI is flagging too much that doesn't need human attention - your thresholds may need raising.
By the end of Crawl, you should be able to answer:
Walk
Your baselines are established. Now you're measuring quality - not just volume. This is where you start looking at accuracy, consistency, and calibration.
Moderator accuracy (QA sampling)
What percentage of moderator decisions are correct when reviewed a second time? Enable Checkstep's QA feature to route a percentage of decisions to secondary review. This gives you an accuracy score per moderator.
Moderator consistency
When two moderators review similar content, do they reach the same decision? Inconsistency is normal early on - the question is whether it's decreasing over time as your guidelines mature.
Appeal overturn rate
What percentage of user appeals result in the original decision being overturned? A high overturn rate suggests your initial decisions (human or automated) are getting it wrong too often.
Threshold calibration analysis
Look at your confidence score distributions by model. Are your thresholds aligned with where each model's accuracy drops off? Different models have different confidence profiles - a 70% from AWS Rekognition means something different than a 70% from an LLM.
Policy violation breakdown
Which policies are triggered most often? Which have the highest enforcement rates? This tells you where your platform's content risks are concentrated - and where to focus policy refinement.
Flagged-for vs. enforced-for gap
Content can be flagged by one policy but ultimately enforced under a different one. Tracking this gap reveals whether your AI labels are accurate or whether moderators are frequently re-categorising violations. A large gap means your strategies need attention.
By the end of Walk, you should be able to answer:
Run
Your operation is stable and accurate. Now the metrics shift from operational to strategic - measuring trends, efficiency, and ROI. Your Reporting Dashboard becomes a management tool, not just an operational one.
Harm trend analysis
Is harmful content on your platform increasing or decreasing over time? Look at this by policy category. A spike in hate speech might indicate a real-world event or a new vector your policies don't cover yet.
Repeat offender rate
What percentage of enforcement actions are against users who've been actioned before? A high repeat rate suggests your escalation path (warn → suspend → terminate) may not be aggressive enough - or that bad actors are creating new accounts.
Automation rate over time
Track the percentage of total moderation decisions made automatically versus by humans. This should trend upward as you refine thresholds, improve labels, and deploy ModBot. This is your primary efficiency metric.
Compliance SLA adherence
Are you meeting regulatory timelines for user notifications, appeal responses, and transparency reporting? The DSA has specific SLAs - track whether your team is meeting them consistently.
Cost per 1,000 pieces of content
Combine your AI credit usage, moderator time, and tooling costs into a per-content metric. This is what leadership and finance need - a clear cost picture that ties moderation to operational efficiency.
At the Run stage, you should be able to answer:
Summary: metrics at each stage
A quick reference for which metrics matter when.
| Metric | Crawl | Walk | Run |
|---|---|---|---|
| Daily incident volume | ✓ | ✓ | ✓ |
| Auto-enforcement rate | ✓ | ✓ | ✓ |
| Human review queue size | ✓ | ✓ | ✓ |
| Average handling time | ✓ | ✓ | ✓ |
| Moderator accuracy (QA) | · | ✓ | ✓ |
| Moderator consistency | · | ✓ | ✓ |
| Appeal overturn rate | · | ✓ | ✓ |
| Threshold calibration | · | ✓ | ✓ |
| Policy violation breakdown | · | ✓ | ✓ |
| Harm trend analysis | · | · | ✓ |
| Repeat offender rate | · | · | ✓ |
| Automation rate over time | · | · | ✓ |
| Compliance SLA adherence | · | · | ✓ |
| Cost per 1,000 pieces | · | · | ✓ |
A note on benchmarks
There is no universal "good" number for most Trust & Safety metrics.
The right auto-enforcement rate for a children's gaming platform looks very different from the right rate for a B2B marketplace. What matters is that your metrics are moving in the right direction, your thresholds are calibrated to your specific content mix, and your team has the context to interpret what they're seeing.
Checkstep's team has worked with platforms across gaming, social, marketplaces, streaming, and dating. Your account manager can share relevant benchmarks from comparable platforms to help you contextualise your numbers - reach out directly for a metrics review or to benchmark your performance against industry data.