While I’m looking at this for an actual use case, I should note that orchestrator will continue to support mysql as backend (and I/we will continue to use it like that…).

You’re right about the architectural changes that would be required of orchestrator. We discussed some of those in the past: sharing work between orchestrator nodes; taking leadership; getting quorum agreement for failover; operation locks.

I do believe with proper layering most of these changes would not have to complicate existing code, which would mostly be ignorant of the change. But — ongoing effort, so we’ll see.

If I were to do what you want I guess I would keep state in memory with a very basic setup on disk of the boxes to monitor (not their state) and if I needed HA I would exchange “config” and/or state with neighbours etc. Basically like networking daemons exchange routing information. The idea sounds simple, but having seen and knowing the current code that’s a major change in the way things work and quite a risky undertaking though maybe it would allow “instant” recognition of failure scenarios and therefore be “better” than now. Most people don’t really appreciate the complexity of what orchestrator does, the situations it handles and in practice a new in-memory based orchestrator server would also be pretty complex. It might be an interesting path to take but few people probably have time for such an adventure.