orchestrator 3.0.2 GA released: raft consensus, SQLite

orchestrator 3.0.2 GA is released and available for download (see also packagecloud repository).

3.0.2 is the first stable release in the 3.0* series, introducing (recap from 3.0 pre-release announcement):

orchestrator/raft

Raft is a consensus protocol, supporting leader election and consensus across a distributed system.  In an orchestrator/raft setup orchestrator nodes talk to each other via raft protocol, form consensus and elect a leader. Each orchestrator node has its own dedicated backend database. The backend databases do not speak to each other; only the orchestrator nodes speak to each other.

No MySQL replication setup needed; the backend DBs act as standalone servers. In fact, the backend server doesn’t have to be MySQL, and SQLiteis supported. orchestrator now ships with SQLite embedded, no external dependency needed.

For details, please refer to the documentation:

• orchestrator/raft: overview
• orchestrator/raft vs. shared backend DB setup, comparison

SQLite

Suggested and requested by many, is to remove orchestrator‘s own dependency on a MySQL backend. orchestrator now supports a SQLite backend.

SQLite is a transactional, relational, embedded database, and as of 3.0 it is embedded within orchestrator, no external dependency required.

orchestrator-client

orchestrator-client is a client shell script which mimics the command line interface, while running curl | jq requests against the HTTP API. It stands to simplify your deployments: interacting with the orchestrator service via orchestrator-client is easier and only requires you to place a shell script (this is as opposed to installing the orchestrator binary + configuration file).

orchestrator-client is the way to interact with your orchestrator/raft cluster. orchestrator-client now has its own RPM/deb release package.

You may still use the web interface, web API ; and a special --ignore-raft-setup keeps power at your hand (use at your own risk).

State of orchestrator/raft

orchestrator/raft is a big change:

• In the way it is deployed
• In the way it is operated
• In the high availability it provides
• and more

This is why it has been tested in production for a few months.

orchestrator/raft now runs in production at GitHub ; we’ve decommissioned the “old” orchestrator setup having run both in parallel for a while. It drives our failovers and deploys over three data centers.

We are using MySQL as backend to our orchestrator cluster. We will introduce more staging tests for SQLite-based setups.

Roadmap

There’s much to do, and we chose to release a version that has a way to go. We expect:

• Dynamic raft cluster join/leave operations (right now the cluster is static and configuration based)
• New nodes joining the cluster to auto-populate data from the cluster. This is actually a built-in feature to the hashicorp/raft library that we use, however we intentionally did not use this functionality, and expect to re-introduce it. At this time, adding a newly provisioned node to the cluster requires a backup/restore or dump/load of DB data from an existing node.
• Partitioning of probe tasks across nodes
• Various thoughts on proxy integrations
• More…

We will focus on making operations simpler, and of course keep stability and reliability at highest priority.

orchestrator tutorial

In two weeks time I will be presenting the practical orchestrator tutorial, a 3 hour practical walkthrough on deployment, configuration, reasoning and more.