This was a rather interesting event. In general, changing the IP address (even the loopback address) shouldn't have caused it from the BGP perspective. For example, if you were to change the IP address of BGP enabled router that has multiple BGP sessions, all other routers tore down the sessions to it, and withdrew the prefixes. BGP reconverge events take time. However, less than this took (90+ minutes and then a few more hours until __full__ recovery).
This seems like one of the events in which they changed IP on Route Reflector routers that were pretty busy, which would cause reconvergence and CPU spikes for all routers that it had sessions with. Also, there was a lot of volatility, as part of which re-advertisements were happening continuously. They also attempted rollback, which caused reverse operation, which triggered reconvergence. The other scenario is doing this change on the SDN controller, which affected all other routers.
If there was a shorter route and you took longer one you are dealing with suboptimal routing :)
However, "is a bit sad in a way" part of sentence is interesting one. Edge services hosted within AWS/Cloudflare/Akamai improved customer experience significantly given that waiting for trans-Atlantic or trans-Pacific latencies is not thing any more.
This comment is spot on. This is asynchronous nature of the computer networks. While it is easy to control the path within smaller or enterprise networks, it is very likely that the reverse path on the internet is going to be different compared to forwarding path. Following that, the real challenge is, in case that you are dealing with some issue, to detect where the issue is, given that it might be happening on the reverse path that you don't have visibility into.
Thanks for providing quite good context in the post itself!
I was just about to ask about differences with Tailscale, which is solving a lot of the challenges outlined in the post, but you answered it:
"First off, Netmaker is super fast because it can use kernel WireGuard. There are some other WireGuard-based solutions like Tailscale, but they use userspace WireGuard, which is much, much slower."