Major Incident War Room

HealthNet Regional Provider · Executive Status & Coordination

Active Majors
1
INC-M-091
Time to Exec Brief
18m
Target: <30m
Comms SLA
100%
3/3 on time
Accounts Affected
12.4K
Notified
Recurrence
0
14mo since last
Review Due
2d
Post-major

INC-M-091: EMR Read-Only Access Failure

Major Declared
14:22 UTC · 0m
Auto-escalated from SEV-1 after 3min SLA breach
War Room Activated
14:25 UTC · +3m
Incident Commander: Sarah Chen, Tech Lead: Marcus Rivera
Customer Notification Sent
14:30 UTC · +8m
12,400 patient accounts notified via SMS/email (HIPAA-compliant)
Status Page Updated
14:32 UTC · +10m
Public status: "Investigating EMR access issues"
Executive Brief Delivered
14:40 UTC · +18m
CTO, VP Eng, VP Clinical Ops, Legal briefed — board-ready pack sent
Root Cause Identified
15:15 UTC · +53m
Database connection pool exhaustion due to query timeout spike
Service Restored
15:42 UTC · +80m
Full read/write access confirmed across all facilities
Post-Major Review Scheduled
Pending · +2d target
Stakeholder invites sent, decision log prep in progress

Communications Cadence

Customer Notification (T+8m)
12,400 accounts · HIPAA-compliant template · SMS + Email
On time
Status Page Update (T+10m)
Public status: Investigating · Auto-sync enabled
On time
Executive Brief (T+18m)
CTO, VP Eng, VP Clinical Ops, Legal · Board-ready pack
On time
Stakeholder Update (T+30m)
Clinical staff, support teams, regional managers
On time
Resolution Notification (T+85m)
All channels · Service fully restored message
Complete

War Room Roles

IC
Incident Commander
Sarah Chen · @schen
Active
TL
Tech Lead
Marcus Rivera · @mrivera
Active
CO
Comms Officer
Jennifer Wu · @jwu
Active
SC
Scribe
Alex Kumar · @akumar
Active
LG
Legal/Compliance
Rachel Donovan · @rdonovan
Active

Decision Log · INC-M-091

Time (UTC) Decision Owner Rationale Status
14:25 Declare major incident, activate war room Sarah Chen SEV-1 breach >3min, patient care impact confirmed Executed
14:28 Send HIPAA-compliant patient notification immediately Jennifer Wu Legal requirement <15m for PHI-adjacent outages Executed
14:35 Escalate to database team, grant elevated diagnostic access Marcus Rivera Connection pool metrics inconclusive, need query logs Executed
14:48 Deploy emergency connection pool scale-up (2x capacity) Marcus Rivera Root cause confirmed, tested in staging, low risk Executed
15:00 Defer post-incident review to Monday +2d Sarah Chen Key stakeholders unavailable Friday, need full attendance Pending
15:42 Close major incident, transition to post-major review phase Sarah Chen Service fully restored, all facilities confirmed operational Executed

Customer Impact by Facility

War Room Duration Trend (Last 6 Majors)

Status Page Sync

100%
Updates on schedule

Role Fill Rate

100%
5/5 roles active

Recurrence Watch

0
Same root cause
Last DB pool major:
14 months ago (INC-M-062)

Executive Summary · INC-M-091

Incident: EMR read-only access failure affecting 12,400 patient accounts across 8 regional facilities.
Impact: 80-minute outage preventing clinical staff from updating patient records. Read access maintained; no data loss. Zero HIPAA violations.
Root Cause: Database connection pool exhaustion triggered by query timeout spike in billing reconciliation job.
Resolution: Emergency pool capacity scale-up (2x) deployed at T+48m. Service fully restored T+80m. All facilities confirmed operational.
Customer Comms: HIPAA-compliant notifications sent T+8m (100% SLA compliance). Status page updated T+10m. Executive brief delivered T+18m.
Next Steps: Post-major review scheduled Monday +2d. Billing job refactor & connection pool auto-scaling PRs in progress. Board appendix auto-generated.

Post-Major Reviews

INC-M-091
Due 2d
EMR Read-Only Access · Scheduled Mon 10am
INC-M-088
Complete
Portal Login Timeout · Reviewed, 4 action items closed
INC-M-084
Complete
Billing Batch Failure · Reviewed, systemic fix deployed
HealthNet Major Incident Management Policy v3.2 · Last updated Nov 2024
Incident Response Playbook HIPAA Compliance Guide War Room Procedures