Incident Response

Incident response is the discipline of handling a security incident from the moment something looks wrong to the moment the lessons are written down. It is a process, not a tool, and the process exists because the decisions are hard to make well while under pressure.

The phases

Preparation  ->  Detection & Analysis  ->  Containment  ->  Eradication  ->  Recovery  ->  Lessons Learned
                          ^                                                       |
                          +-------------------------------------------------------+

The loop back matters: containment often reveals a second foothold, which sends you back to analysis.

Preparation is everything done before: logging that is actually collected, an asset inventory, known-good baselines, a contact list, and a decision made in advance about who is allowed to pull the plug.
Detection and analysis turns an alert into an understood event: what happened, when it started, what is affected, and whether it is still happening. Scope before you act.
Containment stops the spread without destroying what you need to understand it. Short-term containment is often network isolation; long-term is a rebuilt, patched system ready to take over.
Eradication removes the foothold: the implant, the account, the modified firmware, the vulnerability that allowed it.
Recovery brings systems back and watches them closely, because the first sign that eradication failed is a reinfection.
Lessons learned is the phase that gets skipped, and it is the only one that changes the outcome of the next incident.

Volatility ordering

Evidence disappears at different rates, so collection has an order. Fastest-fading first:

CPU registers, cache  ->  RAM, network connections, running processes
  ->  temporary files  ->  disk / flash  ->  remote logs  ->  archived backups

Pulling power on a compromised machine destroys everything above "disk" instantly. That is sometimes the right call and it should be a deliberate one.

Chain of custody runs alongside: who collected what, when, with which tool, and what the hash of the acquired image was. Without it, findings may be technically correct and legally worthless.

What changes on embedded and OT

The standard playbook assumes things embedded systems do not have:

There is no agent and no EDR. Detection has to come from the network, because the device cannot report on itself.
There may be no logs at all, or a ring buffer in RAM that a reboot erases. The instinctive "reboot it and see" destroys the evidence.
You often cannot take the system offline. In OT, stopping the process may be more damaging than the incident. Containment becomes network segmentation, not shutdown.
Acquisition means desoldering. Getting a forensic image of a device may require pulling the flash chip with a soic-clip and flashrom, which is destructive to availability and sometimes to the device.
Baselines rarely exist. Without a known-good firmware image to compare against, deciding whether a device is modified is genuinely hard. Comparing a suspect dump to the vendor's published image is often the only route.

The practical consequence is that passive network capture is the primary evidence source for this class of system, and that logging exported off the device before an incident is worth more than any tool applied during one.

Indicators

An indicator of compromise is an observable that suggests a specific intrusion: a hash, a domain, an IP, a filename, a registry key, a distinctive protocol pattern. They are useful and shallow. An adversary changes a hash or a domain in seconds, whereas the behaviour (beaconing on a fixed interval, DNS names with high-entropy labels, a controller receiving writes from a host that has never written before) is far more expensive to change, and is what detection should target.

Pitfalls

Acting before scoping. Isolating one machine tells the intruder they are seen and often costs you the rest of the picture.
Analysing on the live system. Every command you run changes it. Acquire, hash, and work on the copy.
Assuming the first thing found is the entry point. It usually is not.
Declaring victory at recovery. Without the lessons phase, the same incident recurs.

What it pairs with

wireshark is the workhorse for the network evidence, and wazuh covers centralised log collection and alerting. On OT specifically, read this alongside ot-security, where availability constraints reshape every containment decision.