Before an operations desk trusts any workflow that spends money, pages people, or touches customer-facing systems, it inspects the workflow. When the workflow under inspection runs through a shared chat channel — a bot posting events, staff issuing commands, actions logged as they happen — the inspection takes a well-defined shape. This review checklist covers that shape in ten checks, ordered from identity outward to governance. This review draws its structure and supporting evidence from the ChatOps entry of the Ops Atlas, and it is written to be used: each check states what to inspect and the condition that counts as passing.
Come prepared with the paperwork. The review needs the catalogue of permitted actions as currently documented, the written permission policy, proof of where records are stored, and a recent extract of refused requests. Anything that cannot be produced on the spot becomes a finding in its own right, and a predictive one: missing paperwork rarely travels alone. Invite the bot’s owner plus two or three of the channel’s most active daily users; their disagreements about how the system actually behaves are often the most accurate findings of the day.
Phase One: Identity and Origin — Who Is Really Asking?
The first phase asks a single question in three ways: when a request arrives, can you prove where it came from and who sent it? Everything downstream — permissions, approvals, records — inherits its strength from this foundation, which is why the review opens here rather than with the feature list.
Check 1: Proof of origin is enforced on every platform in use.
Inspect the gateway configuration. Passing means each connected platform’s verification is active — Slack requests validated by signature with stale replays rejected, Teams requests validated as signed tokens, and any platform whose authentication covers less than the full payload flagged with compensating controls. Failing looks like “we trust messages because they appear in the channel,” which is the configuration incidents are made of.
Check 2: Authorisation binds to immutable identity, never display names.
Sample ten permission decisions from the most recent month of activity. Passing means every one was granted against a permanent account identifier. Anywhere a readable name — a person’s handle or a channel’s title — participates in an access decision is a finding, because names change, get reused, and can be set at will by members.
Check 3: Channel permissions reference channel IDs.
Rename a channel in a test workspace and see whether permissions follow. Passing means they do, because the policy references internal identifiers. A setup that breaks when a channel is renamed will also break, less visibly, when someone creates a new channel under an old name.
Phase Two: Action Discipline — What Can the ChatOps Bot Do?
Check 4: The action catalogue is complete and owned.
Ask for the current list of everything the bot can do, and for each entry, the named owner and the business reason. Passing means the list exists, matches reality, and has no orphaned entries. An unowned command set grows until it contains abilities nobody remembers approving — a pattern so common it should be presumed until proven otherwise. Count the entries too: a catalogue that has doubled in a year without a matching governance conversation is not evidence of success, whatever the usage charts say, and the first re-review usually trims as much as it adds.
Check 5: Actions are graded, and the grades are enforced.
Review the tier assignments: announcements that only inform, read-only lookups, reversible changes, and high-impact operations. Passing means reversible writes require the appropriate role within the appropriate channel and leave log entries, while the highest tier requires a second person’s approval before anything executes — and a documented emergency override exists for genuine incidents, itself logged and reviewed.
Check 6: No free text reaches a shell.
Trace the full execution path of every single command. Passing means each one maps to a fixed, tested operation. Anywhere user-typed text is concatenated into a shell invocation is an automatic fail; that design converts your channel into a command prompt for whoever can type, which is the anti-pattern the whole allow-list concept exists to prevent.
Phase Three: Records and Attention — Can You Prove and Breathe?
Check 7: The audit destination is outside the chat platform.
Locate where command records go. Passing means an append-only store the bot writes to and nobody edits, separate from chat history. A transcript fails this check by nature: it can be revised, expires under retention policies, and cannot be queried the way an auditor or an incident commander needs.
Check 8: Secrets are managed as secrets.
Inventory every token and webhook URL the integration uses. Passing means storage in a proper secret manager, a rotation schedule that has actually fired, and zero credentials pasted into channel topics, runbooks or code comments. Webhooks deserve particular suspicion — each one is a bearer credential that lets its holder post as your bot, and history shows they are most often discovered in the wild precisely where this check looks for them: pasted into runbooks and channel descriptions by well-meaning colleagues.
Check 9: The notification budget is measured, not assumed.
Pull the trailing two weeks of bot messages and compute the share that asked a human to act. Reference teams have measured near a quarter of messages requiring action against targets around half; whatever your numbers, passing means someone owns the ratio, related alerts are grouped, and load per shift is tracked. An unread channel is a failed channel regardless of its engineering. While computing the ratio, note the grouping: a hundred individually announced events that could have been three summary posts is the clearest possible evidence that the desk is spending attention it has not budgeted.
Check 10: Reviews happen on a calendar.
Inspect the governance rhythm: weekly sampling of refusals, quarterly review of the action catalogue’s scope, and a standing owner for the bot itself. Passing means the meetings exist on real calendars, actually happened, and produced recorded changes. Reviews that exist in a policy document but not in a calendar are findings, not mitigations.
Scoring the Review and Acting on It
Score conservatively: a check passes only with evidence, never with assurance. Then act on the score along these lines:
- Ten of ten. The channel is ready for more consequential work; new catalogue entries still enter through the same grading.
- Seven to nine. A normal first-review result. Fix identity and logging findings first — every other control depends on them.
- Six or below. Restrict the bot to read-only operation until the foundations hold; a well-behaved informer remains useful while you repair.
Whatever the score, write the findings down with dates and owners. A review that ends in conversation rather than a filed document will be re-litigated in three months with nothing settled.
And when the review surfaces gaps in the wider operations stack, the same inspection logic extends naturally to AIOps, automation and observability programs, where identical questions about identity, approval and evidence apply.
Making the ChatOps Review a Quarterly Habit
One pass through this checklist catches today’s gaps; the habit catches next quarter’s. Drift is the enemy — new commands appear, platforms revise their interfaces, staff rotate, and the tidy setup of March becomes the loose one of September. Book the re-review before you finish the first one, rotate the reviewer when you can, and keep the completed scorecards with their dates and scores: few artefacts demonstrate operational maturity as plainly as a growing file of dated, signed readiness reviews, each one slightly sharper than the last. Teams building careers around this discipline will recognise the broader pattern from a structured DevOps career roadmap and from enterprise training in cloud-native platforms — the tools change; the inspect-then-trust cycle does not. ChatOps deserves the same professional scepticism you would give any other system that acts on your behalf, and this checklist is how you apply it. Run the first review this week, file the scorecard, and put the next one in the calendar before you close the laptop — readiness that is not scheduled is merely intention.