Forge Network AIOps

Incident Preview

PE1–SP1 Provider Link Failure

Correlate provider transport loss with service impact without confusing transport state and VPNv4 control-plane state.

This incident runs in the downloadable Forge Lab. The Online Forge Portal only previews the learning journey.

Get the Forge Lab

The investigation loop

A restored ping is only part of the answer.

Observe → correlate → form a hypothesis → inspect evidence → isolate root cause → remediate → verify recovery.

Use real router output and event records to defend your diagnosis. Explain which signals exposed the failure, how they relate across layers, and why the chosen action addresses the fault. Completion requires the recovery checks to pass.

Before

Healthy provider path

CE1 — PE1 — SP1 — PE2 — CE2
             |
            RR1

IS-IS and LDP are healthy across PE1–SP1; BGP-LS represents the provider link; CE1 can reach CE2.

During the incident

Service symptoms, not the answer

  • Customer reachability fails.
  • A provider link becomes degraded.
  • Recent normalized events increase.
  • VPNv4 sessions may remain established while transport has failed.

Protocols

What to correlate

IS-IS, BGP-LS, LDP, MP-BGP VPNv4, BMP, Kafka

Investigation

Read-only command

show isis neighbor

In the Offline Forge Lab, the guided action opens PE1 or SP1 and runs this approved operational command.

Evidence

What proves the fault

Protocol state, BGP-LS event, LDP state, service reachability, normalized event

Example event timeline

Follow evidence without guessing

  1. IS-IS adjacency down on PE1 and SP1
  2. LDP neighbor loss toward the provider core
  3. BGP-LS link withdrawal observed in normalized events
  4. CE1-to-CE2 reachability failure recorded
  5. Recovery restores the link, protocol state, BGP-LS representation, and service reachability

The guided incident asks the learner to distinguish provider transport loss from a CE static-route failure, then selects the single permitted remediation and validates recovery.

Root-cause reasoning

Prove the smallest fault

The learner must show that the CE-facing links and CE static routes are still available while the PE1–SP1 provider transport relationship has disappeared. This rules out a customer-edge route failure.

Remediation and recovery

Restore only the affected link

The local incident permits its deterministic provider-link reset only. It then checks interface state, IS-IS, LDP, BGP-LS, VPNv4 continuity, CE reachability, and the healthy topology baseline.