Onboarding · Field guide
Common first-week mistakes
What to avoid in your first month with Checkstep - a field guide for new Trust & Safety teams.
Going live with content moderation is the easy part. The first-week mistakes are what separate the teams that nail it from the teams that fight their tools for the next six months.
Every item below comes from a Pro tip, Watch For, or Callout block scattered across the existing Checkstep help articles. They're pulled into one place because the second week is too late to find them.
Print this. Post it where your team can see it during their first month. Each mistake links back to the depth in the help center if you want the long version.
Policy mistakes
Most policy mistakes are about confusing the layers of a policy or treating them as interchangeable. The Checkstep policy has three layers - Description, Internal Guidelines, Rules - and each one does a different job.
Copying policy text into your LLM labels
What happens: The label becomes circular. It tells the model to look for X by describing X. False positives surge and accuracy drops within the first week.
Fix: Write labels as behaviors the LLM can actually detect. "False promises of guaranteed investment returns, deceptive loan offers, unsolicited cryptocurrency promotions" beats "Financial spam" every time. The strong label describes observable content; the weak label describes a category.
Source: Advanced Policy & ModBot - Writing better LLM labels
Treating Strategies and rules as the same thing
What happens: New users add labels to strategies but no rules - the strategies tag content but nothing happens. Or they add rules with labels that aren't on any active strategy - references break silently.
Fix: Strategies live under Settings → Strategies and are configured globally. Rules live inside individual policies. Strategies produce labels; rules decide what to do with them. If you remember nothing else from your first week, remember this distinction.
Setting thresholds once and forgetting them
What happens: A 70% confidence score from AWS Rekognition means something genuinely different from a 70% from an LLM classifier. Setting the same threshold across both models produces unpredictable results.
Fix: Plan a tuning session with your account manager one to two weeks after going live. Your day-one thresholds are starting points, not final settings. Different models need different thresholds; you can't infer the right numbers without observed data.
Source: Building Your Policies - Thresholds & confidence scores
Writing Internal Guidelines as restatements of the public policy
What happens: Moderators (and ModBot) need more context than your users do. If your internal guidelines simply restate the public-facing policy, edge cases land inconsistently and you'll see disagreement between moderators within the first few days.
Fix: Internal guidelines should cover what users don't see: regional exceptions, age-related thresholds, edge cases, escalation criteria, and paired examples (content that looks similar but lands on opposite sides of the line).
AI tuning mistakes
These are about working with the AI rather than against it. The reflex when something's wrong is often to adjust the model. Almost always, the actual fix is upstream.
Writing macro labels instead of micro behaviors
What happens: A label like "political content" can't be matched against actual posts. It's too broad and too interpretive - the LLM produces noisy results because the target is fuzzy.
Fix: Break the macro into micros. "Political content" becomes "debates about election outcomes, partisan attacks on named figures, discussion of specific policy legislation." Each micro is something the model can actually decide on.
Source: Advanced Policy & ModBot - Writing better LLM labels
Adding a new label and immediately creating a rule for it
What happens: You're acting on a signal before you know whether the signal is good. A bad threshold combined with a poorly-defined label produces enforcement actions you'll have to clean up via appeals.
Fix: Add the label first. Let it run for a few days. Review what it's tagging in your dashboard. Refine the label description if needed. Only create a rule once the label is reliably catching the right content.
Source: Advanced Policy & ModBot - Labels you can experiment with safely
Blaming the AI when ModBot makes a wrong call
What happens: The reflex is to dig into model tuning. The actual issue is usually ambiguous policy text - ModBot followed your instructions correctly; the instructions just didn't cover this case.
Fix: When ModBot gets it wrong, read the rationale it wrote. Look at which policy clause it cited. The fix is almost always in the policy or internal guidelines, not the AI. ModBot is a capable but literal team member that does exactly what your written instructions say.
Source: Advanced Policy & ModBot - Tuning policies for ModBot
Operational mistakes
Operational mistakes are usually about not letting the system tell you what it knows. The platform surfaces signals about its own health - ignoring them in week one means cleaning up bigger problems in week three.
Going live with auto-enforcement turned on from day one
What happens: Untested thresholds plus auto-enforcement equals false positives that upset real users. Fixing it after the fact is much harder than just observing first.
Fix: Run in review-only mode for the first week. Watch what the AI flags. Confirm the labels are accurate and the thresholds are sane. Then turn on auto-enforcement, one policy at a time.
Source: Setup Checklist - Step 2 Pro tip
Ignoring the human review queue size signal
What happens: Queue grows faster than your team can clear it. Backlog builds invisibly until someone flags it in week three - and by then there's a triage problem on top of the original tuning problem.
Fix: Queue size is the earliest signal that your thresholds need adjustment. Watch it daily for the first month. A consistently growing queue means you're sending too much for review (loosen thresholds) or you need more moderators.
Treating a high dismiss rate as a productivity number
What happens: If 50% of the items your moderators review get dismissed (no violation found), that's not throughput - that's your team spending half their day on content that didn't need them.
Fix: Above 40–50% dismiss rate in the review queue is a strong signal your thresholds need raising. The right metric is the share of reviewed items that result in real action.
Optimizing thresholds before there's enough data to know what normal looks like
What happens: Adjusting thresholds in week one based on three days of data produces wild swings. You're reacting to noise rather than signal.
Fix: Two weeks of baseline data before any meaningful tuning. Crawl-stage metrics in the first month are about understanding what normal looks like for your platform - not about optimizing.
Compliance and integration mistakes
These are smaller in number but bigger in blast radius. A compliance mistake doesn't inconvenience your team - it creates a regulatory or security incident.
Sharing Transparency Portal redirect URLs
What happens: Redirect URLs contain unique authentication tokens. Sharing one outside the content's author is a privacy and security incident. The token expires after six hours - but a screenshot in a Slack channel doesn't.
Fix: Never share redirect URLs. They're for the content's author only. If you need to discuss a case internally, use the case ID and the moderation dashboard, not the user-facing redirect link.
Turning off webhook retries during a downstream incident
What happens: Your downstream system is down, webhooks pile up, somebody on the team turns retries off to "stop the noise." Real moderation events disappear into the gap.
Fix: Let retries run. If your downstream system will be down for longer than the standard three attempts, contact Checkstep support to request extended retry handling. Turning them off is almost never the right move.
Skipping HMAC payload signing on webhooks
What happens: Without signing, anyone who knows your webhook endpoint URL could spoof a moderation event. It's a credential-free attack surface most teams forget about.
Fix: Enable signing during initial setup. The 24-hour key rotation window means you can rotate keys later without downtime; there's no good reason to skip this.
Not testing the appeals flow before going live
What happens: The DSA and the OSA both require functional appeals processes. If yours breaks the first time a real user submits one, you have a compliance incident on top of an angry user.
Fix: Run a test appeal through the Transparency Portal before going live. Verify it lands in the moderation queue, that your reviewers can see the original content + the appeal note, and that the decision flows back through your downstream notification system.
Source: Setup Checklist - Step 6
How to use this list
Print it. Tape it to a wall your Trust & Safety team will see during their first month of operations. Talk through each item once during your team's first onboarding meeting.
Every mistake above links back to a fuller treatment in the Checkstep help center. The point of this guide isn't to be exhaustive - it's to be visible at the moment when most of these mistakes get made.
If your team hits a mistake we don't cover here, send it to your account manager. The next version of this list should include it.