ISO 27001 · SOC 2 Type II · CMMI Level 3

+84 28 3822 9100
Architecture

Active–active failover on a plant network that was never designed for it

How we get to a 12-second RTO without asking anyone to re-cable a substation.

HĐPhan Hữu ĐứcPrincipal engineer

9 August 202611 min

The textbook answer to plant resilience is a second network. The site answer is that there is one network, it was laid in 2004, and the cost of a second one is a capital project nobody will approve for a system that has not failed yet.

What actually fails

In eleven years of incident logs the failure is almost never the cable. It is a switch that takes itself offline under load, a gateway whose certificate expired, or a server whose disk filled with the logs of the thing that filled the disk. All three are survivable on one physical network if the software stops assuming it owns the machine it runs on.

Twelve seconds

Our target is a twelve-second recovery, measured from the last accepted write. Both nodes ingest continuously; one holds the write lease and renews it every two seconds. Losing the lease is the failover, and because the standby has been ingesting all along there is nothing to catch up on — only a lease to take.

The awkward part is not the failover. It is proving to an operator, on their screen, which node is authoritative right now, without asking them to care.

  • Resilience
  • Networking
  • Deployment

Bring us your hardest site.

A briefing runs ninety minutes with your operations and IT leads in the room. You leave with a signal survey scope and a fixed-price number — no pilot fee, no procurement theatre.