Garanti del Talento

dgt_sre08 – Incident Management

Module Title: dgt_sre08 – Incident Management

Overview:

This module, “dgt_sre08 – Incident Management,” is designed for Site Reliability Engineers (SREs) and professionals who want to learn the main aspects of incident management within an SRE context. The curriculum emphasizes best practices, strategic planning, and post-incident analysis to enhance system reliability and minimize downtime.

The practical sessions follow the Academy’s natural-language approach. During a simulated incident you do not type diagnostic commands: you tell the machine what to investigate (“check error rates and recent deploys for the checkout service in the last 30 minutes”), read the evidence it brings back, decide on mitigation, and verify that it was applied as intended. Knowing the incident lifecycle and how systems fail is what lets you tell a sound hypothesis from a confident but wrong one.

Key Topics:

  1. Introduction to Incident Management:
  2. Understanding the role of SRE in incident management.
  3. Differentiating between incidents, outages, and emergencies.

  4. Overview of incident response frameworks.

  5. Incident Lifecycle:

  6. Phases of an incident from detection to resolution.
  7. Roles and responsibilities within an SRE team during an incident, including the human who directs and supervises the AI agent.

  8. Communication strategies for internal teams and stakeholders.

  9. Documentation and Referencing in SRE:

  10. Importance of documentation and referencing past incidents.
  11. Writing complete runbooks and playbooks in clear natural language that both people and machines can follow.

  12. Using knowledge bases and incident archives.

  13. Testing and Simulation:

  14. Asking the machine to run chaos engineering experiments that test system resilience, within limits you define.
  15. Designing failure simulations and supervising their execution.

  16. Analyzing the outcomes of tests to improve system robustness.

  17. Post-Mortem Analysis:

  18. The significance of producing post-mortems after incidents.
  19. Structuring a thorough and constructive post-mortem report, and checking a machine-drafted timeline against the actual evidence.

  20. Learning from failures to prevent recurrence and enhance processes.

  21. Preventive Measures and Continuous Improvement:

  22. Implementing proactive monitoring and alerting systems.
  23. Strategies for continuous system improvement based on incident learnings.
  24. Encouraging a culture of transparency and accountability within the team.

Module Objectives:

  • Prepare participants to manage incidents in an SRE environment by directing the machine in natural language and verifying every action it takes.
  • Explain the importance of documentation, referencing, and testing in maintaining system reliability.
  • Enable learners to conduct post-mortem analyses that feed into organizational learning and improvement.

Target Audience:

This module is intended for Site Reliability Engineers, DevOps professionals, IT operations staff, and anyone looking to improve their incident management skills within an SRE framework.

The students can push their exercises to the Academy DevOps & SRE Git project. For this module, create a folder with your username as its name in the following subfolder: https://github.com/Garanti-Del-Talento/gdt_academy