Documentation - Intermediate
Building your content policies
Everything you need to configure, structure, and refine your policies inside Checkstep - from your first policy to production-ready moderation rules.
Well-defined policies are the difference between reacting to abuse and preventing it. Teams that get this right see fewer escalations, faster moderator decisions, and consistent enforcement at scale.
Before you begin
This guide picks up where the Getting Started guide leaves off. You should already have your Checkstep account provisioned, your API connected, and at least a basic understanding of what the Policy Dashboard, Moderation Dashboard, and Reporting Dashboard are for.
By the end of this guide, you'll understand how to build a complete policy from scratch, choose the right detection strategies for your content types, set thresholds that balance automation with human review, and connect everything so content flows correctly through your moderation pipeline.
Anatomy of a policy
A policy in Checkstep has three layers. Think of them as moving from "what you believe" to "how the system acts on it."
| Layer | What it does |
|---|---|
| Description | The public-facing text of your policy. This is what end users see when their content is actioned - it appears in the Transparency Portal alongside the removal reason. Write it for your community. Supports full markdown formatting. |
| Internal Guidelines | Additional context, definitions, edge-case examples, and resources for your human moderators. When a moderator encounters a difficult case, this is where they look. If you're using ModBot (advanced), the bot also reads these guidelines to make decisions. |
| Rules | Where policy meets AI. Each rule connects a detection strategy (a model, an LLM prompt, or a keyword list) to a threshold that determines what happens to flagged content - auto-enforce, send for review, or trust. |
Two starting paths
Most customers arrive at policy setup from one of two places. Where you start affects how much configuration work is needed.
Bring your own
You have established community guidelines, an internal policy document, or detailed terms of service that define your content standards. These may need formatting adjustments for Checkstep's markdown editor, but the substance is ready.
Start from templates
You have basic terms & conditions that list categories ("no hate speech, no harassment") without detailed definitions. Checkstep provides reference policy templates across core moderation categories that you can adopt, edit, and make your own.
Many customers fall somewhere in between - they'll import their existing policies for some categories and use Checkstep templates for others. Some customers switching from another platform use onboarding as an opportunity to rewrite or strengthen their policies entirely.
Writing your policy text
The Description field is what your users see. It needs to be clear, specific, and written for your community - not for your legal team.
Checkstep's policy editor uses markdown, which means you can structure your policy text with headlines, lists, bold/italic text, and links. If you're pasting from a Google Doc or Word document, use a rich-text-to-markdown converter to preserve formatting.
What good policy text looks like
Strong policy descriptions share a few traits. They name the specific behaviors that aren't allowed (not just the category), they explain why the policy exists (this builds community trust), and they include examples where possible so users understand the line.
Example: hate speech policy description
Rather than writing "Hate speech is not allowed on this platform," a stronger description names what constitutes hate speech in your context: slurs and dehumanizing language targeting protected characteristics, content promoting hate groups or their symbols, and denial or glorification of violence against groups.
The more specific you are in the description, the better your moderators, your ModBot, and your users can understand where the line is.
Multi-language support
Policy descriptions support multiple languages. If your platform serves a global audience, add translations for each policy so users see removal reasons in their own language. The language selector is available in the policy editor alongside the markdown toolbar.
Internal Guidelines
Internal guidelines are where you put the nuance. Your moderators (and ModBot) need more context than your users do - edge cases, regional considerations, platform-specific exceptions.
Think of internal guidelines as your moderator training manual for each policy category. They should address the scenarios where the right decision isn't obvious.
What to include
Edge cases and exceptions
Does your platform allow self-referential slurs? Is academic discussion of extremist content permitted? Are there age or region-based exceptions? Document these explicitly. Moderators encountering a borderline case will look here - and if you're using ModBot, the bot reads these guidelines to make nuanced decisions.
Real examples
Include examples of content that should and shouldn't be actioned under this policy. Concrete examples reduce ambiguity far more than abstract definitions. Where possible, show paired examples - content that looks similar but falls on different sides of the line.
Escalation criteria
Define when a moderator should escalate rather than decide. Some content categories (CSAM, credible threats of violence, coordinated manipulation) may require immediate escalation to a senior reviewer or legal team regardless of confidence scores.
Understanding Strategies
Strategies are your top-of-funnel scanning layer - the AI models and detection methods that label incoming content before any rules are applied. Checkstep uses the term "strategies" for what you might think of as models or detectors.
There are four core types of strategies. Most accounts will use a combination of these depending on their content types and moderation needs.
Pre-trained models
Ready-to-use models trained on large datasets across core moderation categories. You pick from a menu of labels (such as nudity, violence, hate symbols, alcohol, weapons) and set thresholds - no training or prompt writing needed.
Best for: Visual content categories with well-established detection models. These are your first line of defense for images and video.
Limitation:You can't edit what these models detect - you choose from the available label menu and set thresholds.
Prompt-based (LLM)
The most flexible and widely used strategy. An LLM classifies content based on labels you define with natural-language descriptions. You're essentially prompt engineering - writing descriptions of the behaviors you want to detect.
Best for:Text content, nuanced categories, platform-specific rules that pre-trained models can't cover.
Key insight: Write labels as behaviors to detect, not policy text. "False promises of guaranteed financial returns" is a better label than "Financial spam." Think about what the harmful content actually looks like, not what your policy calls it.
Example-driven
These models learn from examples you provide - typically hundreds of examples per label. Unlike machine learning training (which needs hundreds of thousands of samples), example-driven models work with a manageable dataset that you curate.
Best for:Categories where prompt-based detection isn't precise enough and you have existing labeled data. Particularly strong for platform-specific content patterns.
Note:Setting up an example-driven model is more involved than other strategies - you'll typically work with Checkstep's team during initial configuration.
Keyword-based
Simple string matching against a list of terms you define. Keywords are checked against incoming content and flagged when matched.
Limitation:No context awareness - the word "kill" would match whether someone is discussing a video game strategy or making a threat. Use alongside other strategies, not as your only detection method.
Settings → Strategies, not inside individual policies. This is a common point of confusion - you set up your detection strategies globally, then connect specific labels to policies through rules.Setting up rules
Rules are where you connect your detection strategies to your policies. A rule says: "When this label is detected with this confidence, take this action."
Without rules, your strategies will label content but nothing will happen - no enforcement, no flags, no moderator review. Labels and rules are both required for the system to act.
The label → rule connection
Every strategy produces labels- tags that describe what it detected in a piece of content. When you add a rule to a policy, you're selecting a label from your active strategies and setting a threshold for how the system should respond.
Adding a rule
Inside any policy, go to the Rules tab and click Add New Rule. You'll select a label from the strategies configured in your account, then set the confidence thresholds that determine what happens when that label is triggered.
The order of rules within a policy doesn't matter - all rules for a policy are evaluated against incoming content simultaneously. If content triggers multiple rules, the most restrictive action applies.
Scoping rules to content types
Not every rule needs to apply to all of your content. You can scope rules to specific content types - for example, a "multiple faces" detection rule might only matter for profile photos and be irrelevant for chat messages. Use content type tags sent via your API integration to route content to the right rules.
Thresholds & Confidence Scores
Every strategy returns a confidence score - a number representing how certain the model is that its label applies. Thresholds are where you decide what to do with those scores.
Example configuration: content below 50% confidence is trusted, 50–95% is sent for human review, and above 95% is automatically enforced. These numbers are starting points - your actual thresholds will vary by model and policy.
Why thresholds vary by model
This is one of the most important things to understand about Checkstep configuration: a 70% confidence score from AWS Rekognition means something very different than a 70% from OpenAI's classifier. Each model has its own confidence distribution, accuracy profile, and edge-case behavior.
What this means in practice
AWS Rekognition, for example, tends to be highly accurate at the top of its confidence range but has more fuzziness in the middle. An LLM-based strategy might produce more evenly distributed confidence scores. Setting the same threshold numbers across all models will produce very different results.
This is why Checkstep recommends a baseline configuration session with your account team during onboarding. They know how each model behaves and can set starting thresholds based on your tolerance for false positives versus false negatives.
The automation trade-off
Your threshold configuration reflects a strategic decision about how your platform balances three things:
| Priority | What it means for thresholds |
|---|---|
| Catch everything | Lower thresholds for auto-enforcement, more content sent to human review. Higher volume for moderators, but fewer harmful items slip through. |
| High precision | Higher thresholds for auto-enforcement, trust more content. Fewer false positives, but some harmful content may not be caught. |
| Full automation | Auto-enforce at lower confidence levels, minimal human review queue. Fastest throughput, but highest risk of incorrect enforcement. |
Most platforms land somewhere between catch-everything and high-precision, adjusting by policy category. You'll likely want tighter thresholds for CSAM (miss nothing) and looser thresholds for spam (some false positives are acceptable).
Content type routing
Checkstep handles text, images, video, and audio. Not every strategy needs to run on every content type - and not every rule applies to every part of your platform.
When you ingest content through the API, you can include tagsthat tell Checkstep what kind of content it is (profile photo, chat message, forum post, listing description) and where it came from. Rules can be scoped to these tags, so a rule that blocks nudity in profile pictures doesn't have to fire on every chat message.
AI credit efficiency
Checkstep only runs models on content that's eligible for them. A text-only message won't consume image recognition credits. Checkstep's forecasting tools help you estimate your AI credit usage based on your content mix and the models you've activated - check this during setup so you know what to expect.
The general rule: most accounts need four to five active models. Adding more gives you broader coverage but increases credit usage. Start with the essentials for your content types and add strategies as you identify gaps.
Testing your setup
Before going live with new policies or rule changes, validate your configuration with test content.
Your strategies will label all incoming content regardless of rules. This means you can send test content through the system and observe what labels appear before you create rules that act on them. Use this to verify that your strategies are detecting what you expect.
What to test
Borderline content
Don't just test obvious violations. The real test is borderline content - a message that's edgy but not clearly harmful, an image that's suggestive but not explicit. These edge cases reveal whether your thresholds and labels are calibrated correctly.
Context-dependent content
Checkstep evaluates content in context - a single message might read innocently, but combined with the surrounding conversation or user history, it may be clearly harmful. Test scenarios where context changes the correct moderation decision.
Multi-label scenarios
A single piece of content can trigger multiple labels across multiple policies. Test content that you expect to hit more than one policy to verify that the most appropriate action is taken.
Next steps
Your policies are configured and your rules are set. Here's where to go from here.
Monitor your first week
Use the Reporting Dashboard to track incident volumes, auto-enforcement rates, and moderator throughput. Plan a threshold tuning session after 1–2 weeks of live data.
Advanced: ModBot & automation
Ready to take policy automation further? The Advanced Policy Configuration guide covers ModBot setup, LLM label writing, and tuning your policies for AI-assisted decisions.
If you need help fine-tuning your threshold configuration or writing labels for your LLM strategies, reach out to your account manager or Checkstep's support team. The initial calibration is something most customers do collaboratively with Checkstep - you don't need to figure it all out alone.