Incident Response & Contingency

Response & Contingency Focus: A plan written is the only plan that works in crisis.

Every organization will have a bad day. A cyberattack locks its files. A vendor pushes a faulty update. A data center loses power. A building becomes unreachable. The question is never whether something will go wrong. It is whether the organization decided, long before that day, what it would do when it did.

This pillar covers two related disciplines. Incident response is the work of detecting a specific event, containing it, and recovering from it. Contingency is keeping the business running when a problem cannot be fixed quickly: switching to an alternate site, falling back to a manual process, or turning to a backup supplier. Incident response fixes the problem. Contingency keeps the organization alive while the problem is being fixed.

The pillars before this one set up the conditions for it. Test and Evaluation is the gate: it proves a system is fit before it goes live. Sustainment and Maintenance is the rhythm: it keeps the system sound every day after. Incident Response and Contingency is the response: it takes over when something breaks that the rhythm cannot absorb.

This article lays out the framework first, then the governance and discipline that make it work, and closes with a current example of what happens when organizations have, or lack, a plan.

The Call From 7 World Trade Center

On the morning of September 11, 2001, I was two blocks from the World Trade Center when the first plane hit. Some of my staff were working in 7 World Trade Center, a building across the street from the towers.

During the evacuation, one of my supervisors in that building reached me by phone. Building management was telling my group to stay put. I made the call to get my people out, two at a time. That afternoon, 7 World Trade Center collapsed.

I do not tell this story because the decision was heroic. I tell it because of what it reveals about planning. In that moment, the instructions in front of my staff and the right call were not the same thing, and someone had to decide which one to follow. There was no time to convene a meeting or look up a policy. Whoever had the authority to act had to already know they had it.

That is the lesson of this pillar. In a crisis, there is no time to work out who decides. The organizations that respond well are the ones that settled that question in advance: who can declare an emergency, who can override a standing instruction, and who acts when the usual decision-maker cannot be reached.

The examples in this series span decades. The technology in them has changed many times over. The need to settle authority before the crisis arrives has not.

The Response Framework

Every incident, whatever its cause, moves through the same seven stages. The work inside each stage changes with the type of event. The stages themselves do not.

Prepare. Everything done before anything goes wrong: plans written, roles assigned, backups tested, contacts current, drills run. Most of an organization's response capability is built here, long before it is needed.
Discover. Something surfaces: an alert, a failed system, a call from a customer, a warning from a vendor. Discovery is only as fast as the monitoring and reporting channels built during preparation.
Triage. A deliberate judgment on size and impact. Not every discovery is an emergency. Is this critical, touching safety, core operations, regulatory exposure, customer data, or something that cannot be undone? Or is it correctable, something that can be fixed in the next maintenance window, the next release, or the next quarter without disrupting operations? Treating everything as urgent is its own failure. It burns people out, wastes contingency capacity, and teaches the organization to ignore alarms.
Disposition. Critical findings trigger the response plan, and the contingency plan if needed, without delay. Correctable findings are logged, given an owner and a date, and handed to Change Management as part of the normal maintenance rhythm.
Recover. Contain the damage, restore service, and confirm that what was restored is clean and complete. Recovery ends when the business is operating normally again, not when the first system comes back online.
Find the root cause. Restoring service treats the symptom. Root cause analysis finds why it happened, so it does not happen again. The question is not "what broke?" but "what allowed it to break?" The answer is often a process, not a piece of technology: a change that skipped review, a warning nobody owned, a vendor update nobody staged. The depth of the analysis scales with the outage and the process it affected. A brief disruption to a minor system may need a short review. An outage that touched a critical process, customer data, safety, or a regulatory obligation needs a formal analysis, with findings documented, an owner for each corrective action, and a date to confirm the fix worked.
Learn. An after-action review, held while memories are fresh, that asks what happened, what worked, what did not, and what changes. Every lesson, and every root cause, either updates a plan, a procedure, or a control, or it was not really learned.

Change Management and Contingency Drift

Technology changes constantly. Servers move to the cloud, applications are replaced, vendors are swapped, networks are redesigned, and AI tools are added to workflows that used to be manual. Each of those changes is reviewed for cost, security, and performance. Very few are reviewed for what they do to the organization's ability to recover.

That is contingency drift: the slow gap that opens between the recovery plan on paper and the environment it is supposed to protect. A backup job still runs, but it no longer covers the new system. A manual workaround still exists in the binder, but the paper forms it depended on were retired two years ago. A failover site is still under contract, but it was sized for last year's workload. A recovery password is stored on the very system it is meant to unlock. None of these show up until the day the plan is needed, which is the worst possible day to discover them.

The fix belongs in Change Management, the oversight process described in Pillar 6. Every change request should answer one more question before it is approved: does this change affect how we recover? In practice, that means checking a short list:

Are backups still capturing everything this change touches, and can they be restored?
Does failover or the alternate site still work with this change in place?
Do the manual workarounds still match how the business now operates?
Are recovery keys, passwords, and contact lists still accurate and reachable during an outage?
Does this change alter how fast we can respond, or who we depend on to respond?

If the answer to any of these is yes, the contingency plan is updated as part of the change, not afterward. Responsiveness is not a fixed property of an organization. It erodes by default with every change that nobody checked.

Contingency Planning Essentials

A contingency plan answers one question: how does the organization keep operating while something important is broken? Six elements make the answer real.

Know what matters most. Rank the business functions by how much damage their loss would cause, and decide for each one how long the organization can live without it and how much data it can afford to lose. A trading desk may tolerate minutes. A monthly reporting system may tolerate days. These two numbers, the maximum tolerable outage and the maximum tolerable data loss, drive every other decision in the plan.
Have somewhere else to go. An alternate site, a cloud region, a second data center, or the ability to work remotely. The alternate must be able to carry the load that matters most, not just exist on a contract.
Have a way to work by hand. Manual workarounds for the critical functions: paper forms, phone trees, standalone devices, local copies of essential information. When the water utilities in Pillar 2 were locked out of their control systems, the ones that could run equipment by hand kept water flowing.
Know who to call, and how. A communication plan naming who calls whom, in what order, and through which channels, including channels that do not depend on the systems that may be down. Contact lists are kept current and available offline.
Have a second source. Backup suppliers and carriers for anything critical, identified and, where possible, contracted before they are needed.
Prove the backups restore. A backup that has never been restored is a hope, not a plan. Restores are tested on a schedule, and backups for the most critical systems are kept where an attacker who reaches the main network cannot reach them.

The Response Checklist

A checklist keeps people from skipping steps under pressure. But no single checklist fits every event. A ransomware attack and a flooded data center call for very different actions. The answer is a core checklist used for every incident, plus a scenario module chosen for the type of event.

Core checklist: every incident

Confirm people are safe before anything else.
Declare the incident and its severity, using the authority defined in advance.
Open an incident log: times, decisions, actions, and who made them.
Assemble the response team named in the plan.
Notify leadership and the stakeholders listed for this severity level.
Decide whether the contingency plan is triggered.
Preserve evidence before changing anything that might be needed later.
Assess legal, regulatory, and contractual notification obligations, and start the clock.
Communicate with staff and customers on a set schedule, even when there is nothing new to report.
Confirm recovery is complete and clean before declaring the incident closed.
Conduct root cause analysis, scaled to the severity of the outage and the importance of the affected process.
Hold the after-action review and assign every follow-up to an owner.

Scenario modules: what changes by event type

Event type Typical owner What changes Watch out for

Cyberattack or ransomware CISO Isolate affected systems; involve legal Restoring too early

counsel and, where appropriate, law reinfecting: paying a

enforcement t; restore only from ransom without Legal

backups confirmed clean guidance

Faulty update or change Head of IT Stop the rollout; roll back or apply the Recovery that requires

Operations, with vendor's fix; coordinate with the vendor hands on every

the Change machine;

Advisory Board recovery keys stored on the affected systems

Vendor or supplier failure Vendor Mgmt Switch to the backup supplier Discovering that the

or procurement or manual workaround; backup supplier was

lead invoke contract terms never actually

contracted; a failure

at the vendor's own

third-party supplier, which the prime vendor still owns

Facility loss Facilities or Move people and operations Plans that assume the

corporate security to the alternate site; building, the phones,

lead account for every person or the network will still work

Network or carrier outage Network operations Fail over to the secondary A "backup" carrier

lead carrier or connection; that shares the

shift to offline channels same physical cable route

Data loss or corruption Data owner for the Find the last known good Corruption copied

for that system copy; restore to into the backups

with IT Operations that point; reconcile before anyone noticed what was lost since

AI system misbehavior Business owner of Switch the system off or No off switch, or

the AI system, with to a safe mode; fall back no one left who

the CISO to the manual process; knows the manual

preserve the inputs and outputs process

Loss of key people Human Resources, Activate designated alternates; Knowledge that lived

with the affected rely on written procedures only in one person's

business unit head

Every module needs its own owner. The person best placed to lead a response depends on what failed. The executive who should lead the response to a lost building is rarely the one who should decide whether to switch off a misbehaving AI system. The roles above are typical, not prescriptive. Each organization appoints its own owner for each module, by name, with a named alternate, plus the call tree for that type of outage: the specific stakeholders affected by it, who must be notified, in what order, and how. A facility loss calls on building management, local emergency services, and every employee on site. A misbehaving AI system calls on its business owner, the data team, and the customers it serves. All of it is recorded in the plan. What matters is that the decision is made in advance. When an event begins and nobody knows whose it is, the first and most valuable hour is lost to the question of who is in charge.

The modules are written in advance, stored where they can be reached during an outage, and updated through Change Management whenever the environment changes.

The Governance Behind the Response

In a crisis, governance comes down to four decisions that must be settled in writing before anything goes wrong.

Who can declare an incident. Someone must have clear authority to say "this is an incident, at this severity," and set the response in motion. If that authority is unclear, the first hour is spent debating whether there is a problem.

Who can trigger the contingency plan. Moving operations to an alternate site, shutting down a system, or switching to manual processes costs money and disrupts the business. The person authorized to make that call, and the conditions under which they should, are named in advance.

Who acts when leaders cannot be reached. Incidents happen at night, on holidays, and while executives are on airplanes. Every critical role has a named alternate, and the alternates know they are alternates. The plan also says when a person on the ground may act on their own judgment rather than wait for instructions that may never come.

Who notifies the outside world. Regulators, customers, partners, and insurers often must be told within fixed deadlines. In the European Union, personal data breaches generally must be reported to regulators within 72 hours. In the United States, publicly listed companies must disclose a material cybersecurity incident within four business days of deciding it is material, and healthcare organizations have their own breach notification rules. Organizations operating across borders face several clocks at once. Knowing which apply, and who is responsible for each, cannot wait until the incident.

These decisions sit inside the governance structure from Pillar 1. The RASCI for incident response, who is Responsible, Accountable, Supporting, Consulted, and Informed, is written and approved before the first alarm, not drawn on a whiteboard during it.

Vendors: The Call Tree Runs Both Ways

Vendors play a critical role in almost every incident, and they are the most often overlooked part of the plan. Most organizations depend on dozens of them: software providers, cloud platforms, carriers, hardware manufacturers, outsourced support. When something breaks, the vendor is often the only party that can fix it, and sometimes the vendor is the source of the problem.

Telling a vendor that a response plan is active does two things.

It addresses the organization's own needs. A vendor that knows a customer is in an active incident can assign the right engineers, prioritize the case, and bring in specialists, instead of treating it as one more support ticket in the queue.

It protects the vendor's other customers. A vendor hearing the same symptom from one customer sees a support issue. A vendor hearing it from several customers at once sees a pattern: a faulty update, a compromised component, a failing service. That early awareness lets the vendor warn its entire customer base, stop a bad rollout, or publish a fix before the problem spreads. One customer's call can prevent an outage for hundreds of others.

That only works if both sides know who to call and when. Many vendors today have no formal call tree for incidents: no defined list of who must be notified, in what order, within what time, and through which channel when one of their products is involved in an outage. Many customers have no equivalent list for their vendors. The result is that critical information travels through whoever happens to answer the general support line.

The fix is a two-way call tree, agreed in advance and written into the contract:

Customer to vendor: named escalation contacts at the vendor, reachable around the clock, with a commitment to respond within a defined time when a customer declares an incident.
Vendor to customer: a commitment to notify named contacts promptly when the vendor discovers a defect, a breach, or an outage that could affect the customer, without waiting for the customer to notice first.
Shared expectations: what counts as an incident, how severity is described, how updates will be delivered, and who on each side has authority to make decisions.
Joint exercises: for the most critical vendors, taking part in the customer's drills, so the call tree is tested before it is needed.

The call tree also scales with the size of the event. A routine incident is handled between operations teams and the vendor's support engineers. A catastrophic event, one that threatens the organization itself, an entire region, or lives, such as September 11, a major natural disaster, or a cyberattack that halts the business, also calls for executive-to-executive contact: CEO to CEO, alongside the active infrastructure plan. That conversation does what no support ticket can. It commits the vendor's leadership to the customer's recovery, releases resources and priorities that only an executive can authorize, and puts both companies' full weight behind getting critical operations restored. The executive tier is named in advance, just like the operational tier, so that on the worst day nobody is searching for a phone number.

The call tree does not stop at the vendor. Most vendors depend on third-party suppliers of their own: the data center that hosts their service, the cloud platform under it, the component maker inside their hardware, the subcontractor running their support desk. An outage can begin at any of those layers, often out of the customer's sight. That does not shift the responsibility. The prime vendor owns the problem. The customer's contract is with the prime, and the prime is accountable for restoring service, for notifying the customer, and for managing its own suppliers. That means the prime's call tree must reach down to its suppliers, its contracts must flow the same notification and recovery obligations down to them, and it must be able to tell the customer which of its suppliers support a critical service.

A vendor without a call tree is a single point of failure that no backup system can cover. Asking for one, and confirming it works, belongs in every contract for anything critical.

Cloud Suppliers: The Weakest Link in Many Plans

The weakest vendor plans in many industries today involve cloud suppliers. Organizations have moved critical systems to a small number of large cloud providers, and in the process many have quietly assumed that the provider's resilience is their resilience. It is not.

Cloud providers operate on a shared responsibility model. The provider is responsible for keeping its platform running. The customer remains responsible for its own data, its own configuration, its own access controls, and its own recovery. A provider can be fully meeting its obligations while a customer's systems are down, its data is deleted, or its account is locked.

Four weaknesses appear again and again:

Everything in one place. Production systems, backups, and the recovery tools all sit with the same provider, sometimes in the same region. When that region has an outage, or the account is compromised, the backups go down with everything else.
No real call tree. Many customers have no named contact at their cloud provider. Their priority during an outage depends on the support plan they purchased, and they learn about the problem from the provider's public status page along with everyone else.
Untested failover. Plans say systems will move to another region or another provider. Few organizations have actually done it, and fewer have done it recently.
No exit plan. If the provider raises prices sharply, changes terms, ends a service, or suffers a prolonged failure, the organization has no tested way to move its systems and data elsewhere.

The remedies follow directly from the rest of this pillar. Know exactly which critical functions depend on which provider and region. Keep at least one copy of critical backups outside the provider's environment, or at minimum outside the primary account. Test failover on a schedule, not just on paper. Know the support tier, the escalation contacts, and the executive relationship before the outage, not during it. And maintain an exit plan for every critical cloud service, reviewed through Change Management whenever the environment changes. Some regulators now require exactly this. Financial regulators in Europe, for example, expect firms to manage concentration on technology providers and to maintain exit strategies for critical ones. Organizations in every other industry would be wise to hold themselves to the same standard.

The Discipline That Keeps a Plan Alive

A response plan is only as good as the last time it was exercised. Discipline is what keeps it from going stale.

Drills run on schedule, especially when nothing has gone wrong. The longer an organization goes without an incident, the more confident it becomes and the less prepared it actually is. Tabletop exercises, where the team talks through a scenario around a table, cost little and expose gaps quickly. Full exercises, where systems are actually failed over and restored, prove what tabletops can only suggest.

Plans are updated after every exercise, every incident, and every significant change. A plan that still lists people who have left, phone numbers that no longer work, or systems that were retired is worse than no plan, because people will trust it.

Exceptions are named in advance. The plan says when someone may depart from it, as the Discipline pillar requires of every standard. Under pressure, people will improvise. The goal is that the improvisation happens inside boundaries decided calmly, not invented in the moment.

The after-action review is never skipped. Once service is restored, the temptation is to move on. The review is where the organization turns a bad day into a better plan. Skipping it guarantees the same lesson will be paid for twice.

Stakeholder Responsibilities

During an incident, confusion about roles costs more time than the technical problem itself. Every stakeholder knows their part before the alarm sounds.

Stakeholder Responsibility

CIO) / Governance Authority Owns the response plan, names the incident commander

and alternates, authorizes the contingency plan for critical

events, and reports to executive leadership and the board

Incident Commander Declares severity, directs the response team, keeps the

incident log, and decides when the incident is closed

Change Advisory Board Requires a recovery-impact check on every change,

and approves emergency changes during an incident

Engineering Contains the problem, restores systems, confirms

recovery is clean, and fixes root causes afterward

Business Unit / Operations Runs manual workarounds, sets recovery priorities for

its functions, and confirms when operations are back to normal

Legal / Compliance Determines notification obligations and deadlines, preserves

evidence, and coordinates with regulators and law enforcement

Communications Delivers regular updates to staff, customers, and the public

through channels that work during an outage

Procurement / Contracting I

(in government, the Contracting nvokes vendor obligations and backup supplier agreements,

Officer and Contracting Officer's and makes sure contracts require vendors to report incidents promptly

Representative, or CO/COR)

Vendors and Suppliers Maintain a two-way call tree; notify customers promptly of

defects, breaches, or outages; prioritize customers in an active

incident; warn the wider customer base when a pattern appears;

and take part in joint exercises

Support / Help Desk Serves as the first point of discovery, escalates by severity,

and keeps users informed

Data Center / Network Staff Execute failover and restores, and keep alternate sites and

secondary connections ready

Finance Funds alternate sites, backups, and exercises, and maintains

emergency spending authority

Then and Now

Contingency used to be physical. Phone trees on laminated cards, tapes driven to an offsite vault, a second trading floor kept ready across the river, binders of procedures on a shelf. Recovery was slow, but everyone could see the plan.

Today much of it is automated and invisible. Data replicates to other regions in seconds, systems fail over on their own, and cloud providers promise high availability. That is real progress, and it hides a new risk: organizations assume the automation will cover them and stop asking what happens if it does not. When the cloud provider, the identity system, or the security tool itself is what fails, automated recovery may fail with it. The tools have changed completely. The need for a tested plan, with named people and a manual fallback, has not.

The AI-Era Parallel

AI systems, from the tools a mid-sized insurer uses to triage claims to the frontier systems some now call super intelligence (SI), bring new kinds of incidents. A model can begin giving wrong or harmful answers without any system going down. Data used to train it can be poisoned. A vendor can change or withdraw a model the business depends on overnight. An attacker can manipulate an AI agent into taking actions no one approved.

The response framework still applies, but each AI system needs three things in place before it goes live: a way to switch it off or into a safe mode quickly, a manual process the business can fall back to, and monitoring that can tell when its behavior has changed. The hardest of the three is often the manual fallback, because once a task has been automated for a few years, the people who knew how to do it by hand have moved on. That is why contingency planning for AI starts on the day the system is approved, not the day it fails.

A Current Example: The Day a Security Update Stopped the World

On July 19, 2024, the cybersecurity company CrowdStrike released a routine content update to its widely used endpoint protection software. The update contained a defect. Within hours, Microsoft estimated that about 8.5 million Windows computers worldwide had crashed and could not restart on their own. Airlines grounded flights, hospitals postponed procedures, banks and broadcasters lost systems, and emergency call centers in some areas were affected.

There was no attack and no criminal. A trusted tool, installed specifically to protect these organizations, caused the outage through an ordinary update. Nearly every element of this pillar was tested that day.

Change Management. The update reached machines automatically. Few customers had any say in when it arrived or a way to stage it on a small group first. That is a vendor change bypassing the customer's change control entirely, exactly the kind of exposure the recovery-impact question is meant to surface.

Contingency drift. The fix required a person to reach each affected machine and restart it in a recovery mode. Where disks were encrypted, that also required a recovery key. Some organizations found their recovery keys were stored on systems that were themselves down.

Manual workarounds. Organizations that could fall back to paper check-in, manual scheduling, or standalone devices kept serving customers. Those that had automated away their manual processes waited.

Recovery capacity. Most organizations recovered within a few days. Some took considerably longer, and the difference came down to preparation: how many machines could be reached, how many people were available to touch them, and how current the recovery documentation was.

The lesson applies to every industry and every country. The incident did not ask whether an organization was well regulated or well funded. It asked whether the organization had planned for a failure in something it trusted.

Incident Response, In Three Sentences

The response to a crisis is decided long before the crisis arrives.

Authority, plans, and backups that have not been tested are assumptions, and a crisis is the worst time to find out which assumptions were wrong.

And every change to the environment either keeps the recovery plan current or quietly breaks it.

The Failure Mode

The failure mode is not the absence of a plan. Most organizations have one. Nor is it a single bad day. It is the same bad day repeating, because the organization restored service and moved on without finding out why it failed. Most often, though, it is a plan that was accurate the day it was written and has drifted ever since: people who have left, systems that were replaced, workarounds that no longer match the business, and backups nobody has restored. The plan sits on the shelf looking complete, and the organization believes it is prepared. It finds out otherwise on the day it can least afford to.

Where This Leads

Incident Response and Contingency is the seventh pillar of the IT Risk and Cybersecurity Governance Framework because it is where every other pillar is tested under pressure. Governance decides who acts. Risk Identification tells the organization what to protect first. Process Engineering and Test and Evaluation build and prove the recovery mechanisms. Discipline keeps the drills running. Sustainment and Change Management keep the plan current as technology changes.

But a plan is carried out by people. When the alarm sounds, what matters is whether the people on duty know the plan, have practiced it, and can act on it without waiting to be told. That is the subject of Pillar 8, Training and Workforce Development.

This is Pillar 7 of 8 in the IT Risk and Cybersecurity Governance Framework series. Read the full Framework at MVWConsultants.com/framework.