Announcing orchestrator-agent

orchestrator-agent is a side-kick, complementary project of orchestrator, implementing a daemon service on one’s MySQL hosts which communicates with and accepts commands from orchestrator, built with the original purpose of providing an automated solution for provisioning new or corrupted slaves.

It was built by Outbrain, with Outbrain’s specific use case in mind. While we release it as open source, only a small part of its functionality will appeal to the public (this is why it’s not strictly part of the orchestrator project, which is a general purpose, wide-audience solution). Nevertheless, it is a simple implementation of a daemon, such that can be easily extended by the community. The project is open for pull-requests!

A quick breakdown of orchestrator-agent is as follows:

• Executes as a daemon on linux hosts
• Interacts and invokes OS commands (via bash)
• Does not directly interact with a MySQL server running on that host (does not connect via mysql credentials)
• Expects a single MySQL service on host
• Can control the MySQL service (e.g. stop, start)
• Is familiar with LVM layer on host
• Can take LVM snapshots, mount snapshots, remove snapshots
• Is familiar with the MySQL data directory, disk usage, file system
• Can send snapshot data from a mounted snapshot on a running MySQL host
• Can prepare data directory and receive snapshot data from another host
• Recognizes local/remote datacenters
• Controlled by orchestrator, two orchestrator-agents implement an automated and audited solution for seeding a new/corrupted MySQL host based on a running server.

Offline, hard-copy backups aside, Outbrain implements quick backups via LVM snapshots. Some slaves (depending on strength, datacenter etc.) take the role of snapshot servers. These slaves take an LVM snapshot once per day, and keep it open unmounted. At any given time we might have multiple open snapshots on a snapshot server. At a few minutes notice, we are able to restore MySQL to yesterday’s, the day before, two days before, … state. This only takes mounting of the desired snapshot and starting MySQL over the mounted volume.

We’ve in fact put this method to practice in at least one major occasion last year, that proved it to be of significant worth: we were able to fix a compromise to some dataset within minutes of action.

But we also use these snapshots on a daily basis, putting them constantly to practice (hence validating this backup method routinely): we provision new MySQL servers based on those snapshots. A new or corrupted server can be seeded with a snapshot data (preferably from a host in same datacenter). We don’t need to perform a backup now. We can take the one from tonight.

There is some fuss around this: need to verify MySQL not running on target host, clean up the target host, choose the best source host, choose the best snapshot on source host, mount it, deliver the data, accept the data, do post-copy cleanup, start the MySQL service on target host, … Or we might choose to actually do take a fresh, immediate snapshot from a server not configured as snapshot server… We used to do all these by shell scripts, but realized this will not sustain us.

Orchestrator-agent was invented with this particular use case in mind, and orchestrator was extended to support these operations. Orchestrator can act as the controller of a seed/provisioning operation, getting info from/to orchestrator-agent and coordinating two agents to send/receive the data.

The agent still relies on some external commands (via bash) to implement some functionality; to some extent, we have broken down complex shell scripts then wrapped again, now controlled and audited by the agent; but this will slowly change as we move more code into orchestrator-agent itself.

Should you use orchestrator-agent?

If, like us, you use LVM snapshots, at least as a partial form of backup – bingo!

Otherwise, orchestrator-agent can assist in stopping/starting the MySQL service, tailing the error log; simple stuff like that — probably too much fuss for such basic operations.

But it can be easily extended to support other forms of backup (though do note that orchestrator-agent does not and should not perform scheduled backups, merely manipulate them); so if you like to contribute code into orchestrator-agent, please let us know! It would be nice to have a solution that works for multiple use cases.

Get it

orchestrator-agent is released under the Apache 2.0 license, and is free to use.

Pre-compiled, RPM and deb packages can be found on the releases page.