HUB
Sign inSign up

jacmrob/terra-incident-metrics

By jacmrob

•Updated almost 6 years ago

Script for aggregating Terra incident response metrics

Image
0

452

jacmrob/terra-incident-metrics repository overview

⁠Terra Incident Response tracking

Terra's Service Level Agreements (SLAs⁠) define three levels of production incidents: "Blocker", "Critical", and "Minor", and our agreed-to standards for incident response time per incident type. We collect metrics to determine how well our incident response times adhere to our SLAs.

⁠How we track metrics

We track metrics around "Blocker" and "Critical" type issues, and collect the following metrics per incident:

  • time to issue addressed
  • time to issue remediated
  • time to users informed
  • time to post-mortem notes availible

We detect production incidents either manually or via alerts from PagerDuty. For incidents that go through PagerDuty, the incident "begins" when the PagerDuty alert happens. Otherwise, the incident "begins" when a ticket is created on the PROD Jira board. We collect subsequent timestamps as the production ticket and follow up tickets move in Jira, corresponding to the remediation actions taken, as detailed in our SDLC⁠. We use timestamps in these tools to gather metrics:

Metric startMetric end
time to issue addressedPagerDuty alert if issue went through PD, else Jira bug creation timeJira bug created
time to issue remediatedPagerDuty alert if issue went through PD, else Jira bug creation timeJira bug marked as "Remediated"
time to users informedPagerDuty alert if issue went through PD, else Jira bug creation timeJira field "Users Informed" marked true
time to post-mortem completePagerDuty alert if issue went through PD, else Jira bug creation timePost-mortem epic marked as "Postmortem Meeting Complete"

⁠How we collect metrics

Metrics are collected by querying the Jira API for the two tickets associated with each incident: 1) the bug ticket for the actual incident and 2) the epic ticket to track the incident's post mortem.

The bug ticket caputres the following information:

  • incident start time (either via "Pagerduty incident start time" field or ticket creation time)
  • remediation time (time ticket was moved to "Remediated")
  • Incident priority (Blocker or Critical)
  • user contacted time (via "Users Informed" field)

The epic ticket captures the following information:

  • time the post-mortem was scheduled (time ticket moved to "Postmortem Scheduled")
  • time the post-mortem was complete (time ticket moved to "Postmortem Complete")

Parsed metrics are sent to a BigQuery table called "incident_metrics"⁠ in the terra-sla google project. From here, metrics are exported to the adherence dashboard.

⁠Usage

To load a set of metrics for a single incident:

./export_prod_jira.sh <epic ticket id>
# ex) ./export_prod_jira.sh PROD-365

⁠Adherence Dashboard

From the BigQuery metrics, adherence to our SLAs is tracked on this dashboard⁠.

Metrics are uploaded at the end of post-mortems via a Jenkins job⁠. At the post-mortem we'll also reflect on our adherence and opportunities for improvement to achieve a goal 95% adherence.

Tag summary

Content type

Image

Digest

Size

394.7 MB

Last updated

almost 6 years ago

docker pull jacmrob/terra-incident-metrics