ADAM PULSE Root Cause Analysis for Network Outages: How to Find What Actually Failed
When a network outage occurs, the first alert is rarely the whole story. A failed internet connection can trigger alarms for: Firewalls VPN tunnels Switches Access points Servers Phones Printers Cameras Cloud services Remote monitoring agents That does not necessarily mean ten things failed. It may mean one upstream dependency failed and everything below it became unreachable. That is the purpose of root cause analysis, or RCA: Identify the underlying failure that created the visible symptoms instead of troubleshooting every symptom independently.
Short answer
How Do You Find the Root Cause of a Network Outage?
Use this process:
- Define exactly what users are experiencing.
- Determine the scope of impact.
- Build the dependency path from the affected service outward.
- Establish the incident timeline.
- Identify the first abnormal signal.
- Separate upstream failures from downstream symptoms.
- Compare healthy and unhealthy components.
- Review configuration and environmental changes.
- Test competing root cause hypotheses.
- Verify the suspected cause with independent evidence.
- Restore service.
- Continue the RCA until you understand why the failure occurred and how to prevent recurrence.
The objective is not merely to answer: “What device went down?” It is to answer: “What actually caused the business service to fail?”
What Is Root Cause Analysis in Networking?
Network root cause analysis is the systematic process of identifying the underlying condition responsible for a network incident or service degradation. The root cause might be: An ISP failure A firewall configuration change A failed switch DNS failure Power loss A VPN problem Routing Authentication Cloud infrastructure A software defect A capacity problem A physical cable A failed dependency or a combination of factors. Cisco describes network troubleshooting as a process involving the collection and analysis of network events and performance information to isolate problems, while AWS specifically recommends dependency telemetry so teams can determine whether failures originate in services such as DNS, databases, network connectivity, or other external dependencies.
What Is the Difference Between a Symptom and a Root Cause?
This distinction is fundamental. Symptom A visible consequence of the problem. Examples: Users cannot reach the internet VPN is down Access points appear offline VoIP phones disconnected Microsoft 365 is unreachable Packet loss is high Root Cause The underlying failure that produced those symptoms. Example: Symptoms Firewall monitoring lost VPN down Switch unreachable APs unreachable Phones offline Cloud agent disconnected Root Cause Primary fiber circuit failed. If you troubleshoot every alert independently, you waste time. If you identify the root cause, you can manage the incident as one event.
Why Network Monitoring Often Makes Root Cause Analysis Harder
Monitoring platforms are extremely good at detecting state changes. That can create a problem. Imagine a remote site where the ISP fails. A traditional monitoring system may report: WAN unreachable Firewall unreachable VPN unreachable Switch unreachable Access points unreachable Server unreachable Printer unreachable SNMP unreachable Cloud tunnel unavailable Nine alerts are technically correct. Operationally, however: There may be one incident. Operators need the monitoring platform to correlate downstream alerts instead of asking technicians to manually reconstruct the incident.
The First Rule of Network RCA: Find the First Failure
When many things fail together, ask:
Which component failed first?
That often reveals the dependency responsible for everything that followed. Example: 2:13:57 PM — WAN packet loss begins 2:14:02 PM — ISP gateway unreachable 2:14:07 PM — VPN drops 2:14:13 PM — remote switch monitoring fails 2:14:18 PM — access points show offline 2:14:22 PM — server agent disconnects The root cause hypothesis should not begin with the access points simply because six AP alerts appeared. The stronger hypothesis is: WAN failure beginning at 2:13:57 PM created the downstream failures. This is why accurate timestamps and event correlation matter.
Step 1: Define the Actual Problem
Do not begin an RCA with: “Network is down.” Write a precise problem statement. Better: At 8:42 AM, users at the Tampa office lost access to public internet services and the corporate VPN. Local LAN resources remained accessible. Traffic recovered through the secondary WAN at 8:44 AM. That statement already tells you far more. Capture: Location Start time Users affected Applications affected What still works What does not work Whether service is degraded or completely unavailable Whether failover occurred
Step 2: Determine the Blast Radius
The blast radius is the scope of impact. Ask:
One device?
One VLAN?
One building?
One branch?
Several branches?
One carrier?
One cloud service?
Every customer using the same provider?
The blast radius helps eliminate impossible causes. One User More likely: Endpoint Cable WiFi DHCP Local configuration One Site More likely: ISP Firewall Power Core switch Local DNS WAN configuration Multiple Sites Look for shared dependencies: Carrier Cloud DNS Authentication SD WAN controller Central firewall VPN concentrator One Application Everywhere Look at: Application provider Cloud region DNS Authentication API dependency
Step 3: Build the Dependency Path
Modern network RCA should be dependency based. For a typical branch user accessing a cloud application: User → WiFi/AP → Switch → Firewall → WAN → ISP → Internet → DNS → Cloud Provider → Application For a private corporate resource: User → LAN → Firewall → VPN → ISP → Remote Firewall → Server/Application AWS recommends explicitly identifying the external dependencies a workload relies on and monitoring their health because failures in DNS, network connectivity, databases, APIs, or third party services can otherwise be mistaken for failures inside the application itself. The question is:
At which dependency does healthy behavior become unhealthy?
That boundary is extremely valuable.
Step 4: Establish a Precise Timeline
The timeline is often the backbone of the RCA. Record: Last known healthy state First abnormal metric First alert First user report Configuration changes Failover event Carrier acknowledgement Recovery Follow up failures Example: Time Event
10:02:11 Firewall configuration committed
10:04:03 VPN latency rises
10:04:48 Packet loss begins
10:05:02 VPN tunnel drops
10:05:07 Remote application unavailable
10:11:21 Configuration rolled back
10:11:44 VPN restored
That timeline strongly suggests a relationship worth testing.
Step 5: Ask “What Changed?”
One of the highest value RCA questions is:
What changed immediately before the problem?
Check: Firewall policies Routing tables SD WAN policies DNS VLANs Switch configuration Firmware Software Certificates VPN configuration Public IP addresses ISP equipment Cloud configuration Identity policies Authentication Security rules New devices Maintenance activity Google Cloud describes Network Analyzer behavior that attempts to correlate detected network failures with recent configuration changes to help identify root causes. Automatically correlating network degradation with changes made shortly beforehand is one of the highest-value RCA capabilities.
Step 6: Find the Common Dependency
When multiple unrelated systems fail at the same time, look for what they have in common. Example: Zoom fails Teams fails CRM fails Web browsing fails Cloud backups fail Possible common dependencies: ISP DNS Firewall Power If: Zoom fails Teams works CRM works Web browsing works the ISP is much less likely. Think:
What single component could explain every observed symptom?
That is often a better starting point than investigating the loudest alert.
Step 7: Compare Healthy vs Unhealthy Paths
Comparison is one of the fastest troubleshooting methods. Examples: Wired vs WiFi Wired works, wireless fails. Likely area: WiFi/AP/controller Primary vs Backup WAN Backup works, primary fails. Likely area: Primary carrier or handoff Site A vs Site B Site B works using the same application. Likely area: Site A or its network path IP vs Hostname Public IP works, hostname fails. Likely area: DNS Public Internet vs Private Resources Internet works, corporate resources fail. Likely area: VPN/private routing/authentication Comparison reduces the hypothesis space.
Step 8: Test the Failure From Multiple Vantage Points
One monitoring location can lie to you. For example: A server may appear unreachable because the monitoring probe lost connectivity. Test from: Inside the affected LAN Outside the affected site Another branch A cloud monitoring point The firewall itself The ISP gateway The application side Google Cloud's support guidance recommends documenting packet flow and the key endpoints and transformations involved, including VPNs, proxies, and NAT gateways, because generic errors such as “can't connect to server” are insufficient to identify network ownership and root cause.
Step 9: Check the Upstream Layers Before the Downstream Ones
When a child device alerts, ask whether its parent is healthy. Example: AP Offline Before replacing the AP, check: Switch PoE Switch uplink Firewall WAN Power If twenty APs become unreachable simultaneously: Twenty simultaneous AP failures are less likely than: Switch failure Site power WAN failure Monitoring-path failure This is dependency suppression. The upstream failure should become the incident. The downstream failures should become context.
Step 10: Distinguish Correlation From Causation
Two things happening at the same time does not prove one caused the other. Example: Firewall firmware updated at 2 PM. ISP outage began at 2:05 PM. The proximity is interesting. But verify. Look for evidence such as: Interface logs Carrier alarms Routing changes Packet captures Configuration differences Event logs External monitoring Recovery behavior A good RCA uses timeline correlation to generate hypotheses, then tests those hypotheses.
Step 11: Use the 5 Whys
Once you've identified the immediate technical failure, go deeper. AWS describes 5 Whys analysis as repeatedly asking why to move from surface symptoms to the underlying process, configuration, or systemic cause. Example: Incident The branch lost internet.
Why?
The firewall lost its WAN connection.
Why?
The ISP handoff lost power.
Why?
It was connected to a non UPS outlet.
Why?
The carrier equipment was installed after the original UPS plan was completed.
Why?
There is no commissioning checklist requiring carrier equipment to be verified on protected power. Now we have two root causes: Technical root cause Carrier ONT lost power. Process root cause Installation/commissioning controls did not verify protected power. Fixing only the ONT restores service. Fixing the process prevents the next recurrence. Root Cause Is Not Always One Thing Many incidents have multiple contributing factors. Example: Primary internet circuit fails. Normally, users should remain operational. But: Backup circuit bandwidth is insufficient. VPN failover policy is incorrect. VoIP does not migrate. The incident has: Trigger Primary carrier failure. Contributing cause Backup bandwidth deficiency. Contributing cause Incorrect VPN failover configuration. Business impact cause No automated validation of backup service. A useful RCA distinguishes these. Root Cause vs Trigger vs Contributing Factor Trigger The event that starts the incident. Root Cause The underlying condition that allowed or created the failure. Contributing Factor Something that increased probability, severity, duration, or impact. Symptom The observable consequence. Example: Trigger: Utility power interruption Root cause: Network handoff not connected to UPS Contributing factor: No LTE backup Symptom: Office loses internet This produces better corrective actions.
Step 12: Check Whether Failover Worked
Do not stop the RCA after finding the primary failure. If redundancy existed, ask:
Why did this become an outage at all?
If a business has: Primary fiber Secondary cable 5G Starlink SD WAN the failure of one circuit should not necessarily produce a full outage. Check:
Did failover trigger?
How long did it take?
Did DNS continue working?
Did VPN reconnect?
Did voice recover?
Was backup bandwidth sufficient?
Did cloud applications tolerate the public IP change?
Was the backup healthy before the event?
Organizations do not just need to know WAN1 failed; they need to know whether the business successfully continued operating on backup connectivity.
Step 13: Determine the Business Impact
Technical root cause alone is not enough. Translate the failure into:
Who could not work?
Which locations were affected?
Which applications failed?
Were phones affected?
Were transactions affected?
Did backup maintain operations?
What was the duration?
What was the financial or operational impact?
Traditional monitoring often communicates packet loss, latency, interface states, and tunnel status while business stakeholders need to know whether employees can work, customers can reach them, and which locations are affected.
Step 14: Preserve the Evidence
Capture evidence before it disappears. For intermittent incidents especially, record: Timestamps Packet loss Latency Jitter Interface state Gateway availability Public IP status Traceroutes DNS results VPN state Routing table Firewall logs Device CPU/memory Power events Carrier alarms Configuration changes Failover events Cloud provider status Screenshots Ticket numbers User impact Intermittent circuit faults are notoriously hard to prove after the network has recovered, which is why a carrier-ready evidence package matters.
Step 15: Verify the Root Cause
Before declaring RCA complete, ask:
Does the suspected cause explain all major symptoms?
Did correcting it restore service?
Can the evidence independently support the conclusion?
Could another cause explain the same events?
Did the problem recur after remediation?
Use confidence levels where absolute certainty is unavailable. For example: Confirmed Root Cause Physical fiber cut confirmed by carrier. Probable Root Cause Upstream carrier failure based on gateway loss, external monitoring, and multiple nearby customer incidents. Suspected Root Cause Firewall defect based on logs but not reproducible. This is better than pretending every investigation produces certainty.
What Is a Probable Root Cause?
A probable root cause is the explanation best supported by the available evidence when definitive proof is not yet available. Modern AI assisted operational tools increasingly use this model. For example, Google Cloud's incident investigation tooling explicitly generates probable root cause hypotheses together with contextual evidence and recommended next steps rather than presenting every automated conclusion as certainty. That is a useful philosophy for ADAM PULSE as well: Evidence + hypothesis + confidence + next action rather than: AI says this is definitely the problem.
Common Network RCA Patterns
Pattern 1: Everything at One Site Goes Offline Check first: Power ISP Firewall Core switch Monitoring path Do not investigate every child device separately. Pattern 2: Internet Works but Applications Fail Check: DNS VPN Authentication Cloud service Routing Application status Pattern 3: Multiple Sites Fail Simultaneously Check: Shared carrier Central firewall SD WAN DNS Identity Cloud VPN concentrator Pattern 4: WiFi Users Fail but Wired Users Work Check: APs Controller RF Wireless VLAN Authentication PoE Pattern 5: Only Hostnames Fail Check: DNS Pattern 6: Primary WAN Fails but Users Remain Operational The incident may be successfully mitigated. Root cause: Primary ISP Business impact: Low But still investigate redundancy performance. Pattern 7: Primary WAN Fails and Backup Is Up, but Users Still Fail Investigate: Failover policy Routing DNS VPN Bandwidth Application IP restrictions Backup quality
What Are the Most Common Causes of Network Outages?
Common categories include:
ISP failure Power failure Hardware failure Configuration error DNS failure Routing error VPN failure Switching/VLAN problem Wireless issue Cable/fiber damage Capacity or congestion Software/firmware defect Cloud provider outage Authentication failure Security policy Human error Third party dependency failure The correct RCA should identify evidence for the specific incident rather than simply choosing the statistically common answer.
What Should a Root Cause Analysis Report Include?
A useful RCA report should include: Incident Summary
What happened?
Business Impact
Who and what was affected?
Timeline
What happened and when?
Technical Findings
What did telemetry and logs show?
Root Cause
What underlying failure was identified?
Contributing Factors
What made the incident worse?
Detection
How was the incident discovered?
Response
What actions were taken?
Restoration
How was service restored?
Corrective Actions
What changes prevent recurrence?
Owners
Who owns each corrective action?
Due Dates
When will corrective actions be completed?
Example Network RCA Incident Tampa branch internet outage. Impact 28 employees lost primary internet connectivity for two minutes. Business services continued over Spectrum backup with reduced performance. Timeline 8:42:13 AM — Frontier gateway unreachable 8:42:18 AM — WAN1 declared down 8:42:21 AM — VPN tunnel drops 8:42:37 AM — Spectrum WAN2 becomes active 8:43:02 AM — VPN restored over backup 8:44:10 AM — Business applications confirmed available Root Cause Probable Frontier access network outage. Evidence Frontier gateway unreachable Firewall remained healthy LAN remained healthy Spectrum backup remained reachable Public monitoring from multiple vantage points failed against Frontier endpoint Carrier subsequently acknowledged an area incident Business Impact Low due to successful failover. Corrective Actions Continue carrier escalation Monitor recurrence Validate backup bandwidth Review VPN failover time Produce carrier evidence report That is more useful than: INTERNET DOWN — RESOLVED.
Why RCA Should Continue After Service Is Restored
Restoration and root cause analysis are different objectives. Incident Response Get the customer working. Root Cause Analysis Understand why the incident happened and prevent recurrence. The fastest restoration might be: Reboot firewall. But if the RCA ends there:
Why did the firewall require a reboot?
Memory leak?
Firmware bug?
Session exhaustion?
Power problem?
Configuration?
Hardware failure?
Without answering that, the outage may return.
What Is a Bad Root Cause?
Examples of weak RCAs: “Internet went down.” That is the symptom. “Firewall reboot fixed it.” That is the remediation. “Carrier issue.”
Maybe, but where is the evidence?
“Human error.” Too broad. “DNS.”
Which resolver? Why did it fail?
Good RCA is specific enough to drive prevention.
What Is Alert Correlation?
Alert correlation is the process of connecting related monitoring events so they are treated as one incident rather than many independent failures. Example: Firewall lost WAN VPN down Switch monitoring lost APs unreachable Server monitoring lost Correlated result: Primary WAN failure at Branch 17 caused downstream monitoring loss. Monitoring systems should convert events into meaningful incidents before humans are overwhelmed with alerts.
What Is Dependency Mapping?
Dependency mapping records which components rely on other components. Example: ISP ↓ Firewall ↓ Core Switch ↓ Access Points ↓ Users If the ISP fails, monitoring should understand that the downstream unreachable states may be consequences. Dependency mapping is fundamental to automated RCA.
What Is Change Correlation?
Change correlation compares incidents with recent system changes. Examples: Firewall policy change five minutes before outage Firmware upgrade before device crash DNS update before application failures Route change before VPN outage ISP IP change before cloud authentication failure Google Cloud and other modern observability systems increasingly use change information in diagnostic analysis because configuration changes can provide highly relevant causal context.
How Can AI Help With Network Root Cause Analysis?
AI can assist with: Alert grouping Timeline creation Dependency correlation Change correlation Log summarization Pattern matching Probable cause generation Impact assessment Recommended next actions Incident communication Historical comparison AI should not replace verification. A responsible AI RCA workflow is: SIGNALS ↓ CORRELATION ↓ HYPOTHESES ↓ EVIDENCE ↓ CONFIDENCE ↓ HUMAN/AUTOMATED VERIFICATION ↓ ACTION That is much safer than treating a generated diagnosis as fact. The ADAM Pulse RCA Model ADAM Pulse is designed to help operators avoid mentally reconstructing every incident from dozens of independent alerts. The model is: SIGNAL ↓ DEPENDENCY CORRELATION ↓ INCIDENT ↓ PROBABLE ROOT CAUSE ↓ BUSINESS IMPACT ↓ NEXT ACTION ↓ COMMUNICATION ↓ DOCUMENTATION That moves beyond the traditional device → sensor → alert → technician model toward signal → AI correlation → diagnosis → action → communication.
What Should ADAM Pulse Tell the Technician?
Instead of: 37 alerts the goal should be: Incident Tampa Branch Connectivity Degradation Probable Root Cause Frontier primary WAN outage Confidence 94% Evidence Primary gateway unreachable External probes failed Firewall healthy LAN healthy Spectrum backup reachable No recent firewall changes Impact Users transitioned to backup. Voice operational. Bandwidth degraded. Next Action Open Frontier carrier ticket. Owner NOC Follow Up 30 minutes That is the difference between an alerting platform and a network intelligence platform. Network RCA Checklist DEFINE
☐ What exactly failed?
☐ When did it begin?
☐ Who is affected?
☐ What still works?
SCOPE
☐ One device?
☐ One site?
☐ Multiple sites?
☐ One application?
DEPENDENCIES ☐ Endpoint ☐ WiFi ☐ Switch ☐ Firewall ☐ WAN ☐ ISP ☐ DNS ☐ VPN ☐ Cloud ☐ Identity TIMELINE ☐ Last healthy state ☐ First abnormal signal ☐ First alert ☐ First user report ☐ Changes ☐ Failover ☐ Recovery CORRELATE
☐ Which alert happened first?
☐ Which devices share a dependency?
☐ Which failures are downstream symptoms?
CHANGE ☐ Firewall ☐ DNS ☐ Routing ☐ Firmware ☐ VPN ☐ ISP ☐ Cloud ☐ Identity TEST ☐ Local vs remote ☐ Wired vs wireless ☐ Primary vs backup ☐ IP vs DNS ☐ Public vs private ☐ Site A vs Site B EVIDENCE ☐ Logs ☐ Metrics ☐ Traces ☐ Packet loss ☐ Latency ☐ Gateway ☐ Interface status ☐ Config diff ☐ Carrier status ☐ Cloud status VERIFY
☐ Does the cause explain all symptoms?
☐ Did remediation restore service?
☐ Is independent evidence available?
☐ Confidence assigned?
PREVENT ☐ Corrective action ☐ Owner ☐ Due date ☐ Monitoring improvement ☐ Runbook update ☐ Redundancy improvement
Frequently asked questions
What is root cause analysis in networking?
Root cause analysis is the process of identifying the underlying failure responsible for a network incident rather than treating each visible symptom independently.
How do you find the root cause of a network outage?
Define the impact, map dependencies, build the timeline, identify the first failure, correlate downstream symptoms, review recent changes, test competing hypotheses, and verify the suspected cause with evidence.
What is the difference between root cause and symptom?
A symptom is the visible effect of an incident. The root cause is the underlying condition responsible for producing that effect.
Why does one network outage cause dozens of alerts?
An upstream device or service can have many downstream dependencies. When the upstream component fails, monitoring may report every dependent device as unavailable even though they did not independently fail.
What is the first thing to check during a network outage?
First establish the scope: who is affected, which locations or applications are affected, and what still works. That quickly narrows the possible failure domain.
Why is the incident timeline important?
The timeline helps determine which failure occurred first and whether configuration changes or other events preceded the outage.
What does “what changed?” mean in root cause analysis?
It means reviewing configuration, software, firmware, routing, DNS, VPN, ISP, cloud, and security changes immediately before the incident to identify possible causal relationships.
What is dependency mapping?
Dependency mapping identifies which systems rely on other systems, helping operators distinguish an upstream failure from downstream consequences.
What is alert correlation?
Alert correlation groups related monitoring events into a single incident so operators can focus on the likely common cause.
What is the 5 Whys method?
The 5 Whys repeatedly asks why an incident occurred to move beyond the immediate technical failure toward deeper process or systemic causes. AWS uses the method in incident analysis guidance for exactly this purpose.
Can a network outage have multiple root causes?
Yes. An incident may have a trigger, underlying root cause, and several contributing factors. For example, an ISP failure may trigger an outage while an incorrectly configured backup link increases its business impact.
Should RCA continue after the network is restored?
Yes. Restoration gets users working again. RCA determines why the incident occurred and what should change to prevent recurrence.
How can AI perform root cause analysis?
AI can correlate alerts, logs, dependencies, changes, and historical incidents to generate probable causes and recommended actions. Automated findings should still be supported by evidence and confidence rather than treated as guaranteed diagnoses.
What is root cause analysis in networking?
Root cause analysis is the process of identifying the underlying failure responsible for a network incident rather than treating each visible symptom independently.
How do you find the root cause of a network outage?
Define the impact, map dependencies, build the timeline, identify the first failure, correlate downstream symptoms, review recent changes, test competing hypotheses, and verify the suspected cause with evidence.
What is the difference between root cause and symptom?
A symptom is the visible effect of an incident. The root cause is the underlying condition responsible for producing that effect.
Why does one network outage cause dozens of alerts?
An upstream device or service can have many downstream dependencies. When the upstream component fails, monitoring may report every dependent device as unavailable even though they did not independently fail.
What is the first thing to check during a network outage?
First establish the scope: who is affected, which locations or applications are affected, and what still works. That quickly narrows the possible failure domain.
Why is the incident timeline important?
The timeline helps determine which failure occurred first and whether configuration changes or other events preceded the outage.
What does “what changed?” mean in root cause analysis?
It means reviewing configuration, software, firmware, routing, DNS, VPN, ISP, cloud, and security changes immediately before the incident to identify possible causal relationships.
What is dependency mapping?
Dependency mapping identifies which systems rely on other systems, helping operators distinguish an upstream failure from downstream consequences.
What is alert correlation?
Alert correlation groups related monitoring events into a single incident so operators can focus on the likely common cause.
What is the 5 Whys method?
The 5 Whys repeatedly asks why an incident occurred to move beyond the immediate technical failure toward deeper process or systemic causes. AWS uses the method in incident analysis guidance for exactly this purpose.
Can a network outage have multiple root causes?
Yes. An incident may have a trigger, underlying root cause, and several contributing factors. For example, an ISP failure may trigger an outage while an incorrectly configured backup link increases its business impact.
Should RCA continue after the network is restored?
Yes. Restoration gets users working again. RCA determines why the incident occurred and what should change to prevent recurrence.
How can AI perform root cause analysis?
AI can correlate alerts, logs, dependencies, changes, and historical incidents to generate probable causes and recommended actions. Automated findings should still be supported by evidence and confidence rather than treated as guaranteed diagnoses.
Bottom line
The loudest alert is not necessarily the root cause. Good network root cause analysis asks:
What failed first?
What depends on it?
What changed?
What still works?
What evidence supports the hypothesis?
Why did the failure create business impact?
What prevents it from happening again?
The most useful progression is: EVENTS → TIMELINE → DEPENDENCIES → HYPOTHESIS → EVIDENCE → ROOT CAUSE → ACTION And for modern network operations, the practical goal is to perform more of that analysis before a technician has to stare at dozens of dashboards and reconstruct the incident manually. ADAM Pulse is designed to help with that correlation work.