Script for aggregating Terra incident response metrics
452
Terra's Service Level Agreements (SLAs) define three levels of production incidents: "Blocker", "Critical", and "Minor", and our agreed-to standards for incident response time per incident type. We collect metrics to determine how well our incident response times adhere to our SLAs.
We track metrics around "Blocker" and "Critical" type issues, and collect the following metrics per incident:
We detect production incidents either manually or via alerts from PagerDuty. For incidents that go through PagerDuty, the incident "begins" when the PagerDuty alert happens. Otherwise, the incident "begins" when a ticket is created on the PROD Jira board. We collect subsequent timestamps as the production ticket and follow up tickets move in Jira, corresponding to the remediation actions taken, as detailed in our SDLC. We use timestamps in these tools to gather metrics:
| Metric start | Metric end | |
|---|---|---|
| time to issue addressed | PagerDuty alert if issue went through PD, else Jira bug creation time | Jira bug created |
| time to issue remediated | PagerDuty alert if issue went through PD, else Jira bug creation time | Jira bug marked as "Remediated" |
| time to users informed | PagerDuty alert if issue went through PD, else Jira bug creation time | Jira field "Users Informed" marked true |
| time to post-mortem complete | PagerDuty alert if issue went through PD, else Jira bug creation time | Post-mortem epic marked as "Postmortem Meeting Complete" |
Metrics are collected by querying the Jira API for the two tickets associated with each incident: 1) the bug ticket for the actual incident and 2) the epic ticket to track the incident's post mortem.
The bug ticket caputres the following information:
The epic ticket captures the following information:
Parsed metrics are sent to a BigQuery table called "incident_metrics" in the terra-sla google project. From here, metrics are exported to the adherence dashboard.
To load a set of metrics for a single incident:
./export_prod_jira.sh <epic ticket id>
# ex) ./export_prod_jira.sh PROD-365
From the BigQuery metrics, adherence to our SLAs is tracked on this dashboard.
Metrics are uploaded at the end of post-mortems via a Jenkins job. At the post-mortem we'll also reflect on our adherence and opportunities for improvement to achieve a goal 95% adherence.
Content type
Image
Digest
Size
394.7 MB
Last updated
almost 6 years ago
docker pull jacmrob/terra-incident-metrics