Runbook
Published Last updated
Key Takeaway
A documented step-by-step guide enabling trading team members to respond consistently and correctly to specific operational scenarios—system failures, market emergencies, deployment procedures—without requiring specialized expertise or improvisation during time-critical incidents.
What Is Runbook?
A documented step-by-step guide enabling trading team members to respond consistently and correctly to specific operational scenarios—system failures, market emergencies, deployment procedures—without requiring specialized expertise or improvisation during time-critical incidents.
How Runbook Works
Frequently Asked Questions
Why create runbooks if we can just call the expert when something goes wrong?
Because experts aren't always available. When a critical incident occurs at 3 AM in a timezone where all experts are asleep, waiting for callback introduces unacceptable delay. Runbooks enable whoever is on-call to respond immediately. Additionally, even experts make mistakes under pressure. Documented procedures ensure consistent execution regardless of stress levels. Furthermore, creating runbooks forces teams to think through what they actually do and why, often improving procedures in the process. Finally, runbooks enable training new engineers; instead of learning through apprenticeship, they can follow procedures and ask questions.
What scenarios should I document in runbooks for my trading system?
Cover all common failure scenarios: exchange connectivity loss, price feed disconnection, order execution delays, margin liquidation triggers, position tracking errors, system restart procedures. Include deployment procedures: staged rollouts, database migrations, configuration changes. Document security incidents: suspicious activity detection, account access issues, data breach response. Include communication procedures: what notifications to send to which teams. Add edge cases: what to do if procedures don't work, when to escalate. Start with the most frequent and most damaging incidents; gradually expand to less common scenarios.
How should I organize and maintain runbooks?
Organize by scenario type: failure responses, deployment procedures, security incidents. Include clear symptoms enabling problem identification. Use numbered step-by-step procedures avoiding ambiguity. Add decision trees for unusual conditions. Include estimated resolution time and escalation contacts. Version control runbooks in git, tracking changes and enabling collaboration. Review runbooks periodically, updating based on incident experience. Test procedures regularly to ensure steps still work as markets and infrastructure change. Distribute runbooks to everyone on-call, ensuring accessibility during incidents. Update runbooks immediately after incidents, capturing what you learned.
Common Misconceptions About Runbook
Runbooks eliminate the need for experienced engineers during incidents.
Runbooks enable faster response when experts aren't immediately available, but complex incidents still benefit from expert guidance. Runbooks handle common scenarios well; unusual edge cases often require expert problem-solving. The goal is enabling junior engineers to handle routine problems confidently, freeing experts to address complex issues. Runbooks provide structured starting points but shouldn't replace expert judgment. The best runbooks include clear escalation procedures: when to follow procedures versus when to page an expert.
Once I write runbooks, I can stop improving operational procedures.
Runbooks document current procedures, but procedures should continuously improve as you gain operational experience. Each incident reveals opportunities for improvement: perhaps diagnosis was slow because logging was inadequate, perhaps resolution was difficult because automation was missing. The best teams treat every incident as an opportunity to improve runbooks. Additionally, as systems change—new exchanges, new features, new technologies—runbooks must evolve. Runbooks are living documents requiring continuous maintenance, not static documentation.
Detailed runbooks are only for large operations; small teams don't need them.
Even small teams benefit from basic runbooks. A two-person trading team makes mistakes during crises from stress and fatigue; documented procedures reduce errors. When one person is unavailable, the other can respond following procedures rather than being helpless. Runbooks need not be elaborate; even simple bullet-point procedures for common scenarios provide substantial value. Start with the most critical scenarios, gradually expanding coverage. The investment in documentation pays immediate dividends through faster response and fewer mistakes.