Skip to content

# Tag: Open Source

## orchestrator on DB AMA: show notes

MySQL Open Source orchestrator May 26, 2020

Earlier today I presented orchestrator on DB AMA. Thank you to the organizers Morgan Tocker, Liz van Dijk and Frédéric Descamps for hosting me, and thank you to all who participated!

This was a no-slides, all command-line walkthrough of some of `orchestrator`‘s capabilities, highlighting refactoring, topology analysis, takeovers and failovers, and discussing a bit of scripting and HTTP API tips.

The recording is available on YouTube (also embedded on https://dbama.now.sh/#history).

To present `orchestrator`, I used the new shiny docker CI environment; it’s a single docker image running `orchestrator`, a 4-node MySQL replication topology (courtesy dbdeployer), heartbeat injection, `Consul`, `consul-template` and `HAProxy`. You can run it, too! Just clone the `orchestrator` repo, then run:

    ./script/dock system

From there, you may follow the same playbook I used in the presentation, available as orchestrator-demo-playbook.sh.

Hope you find the presentation and the playbook to be useful resources.

## orchestrator: what’s new in CI, testing & development

MySQL dbdeployer docker GitHub Open Source orchestrator testing May 11, 2020May 12, 2020

Recent focus on development & testing yielded with new orchestrator environments and offerings for developers and with increased reliability and trust. This post illustrates the new changes, and see Developers section on the official documentation for more details.

## Testing

In the past four years `orchestrator` was developed at GitHub, and using GitHub’s environments for testing. This is very useful for testing `orchestrator`‘s behavior within GitHub, interacting with its internal infrastructure, and validating failover behavior in a production environment. These tests and their results are not visible to the public, though.

Now that `orchestrator` is developed outside GitHub (that is, outside GitHub the company, not GitHub the platform) I wanted to improve on the testing framework, making it visible, accessible and contribute-able to the community. Thankfully, the GitHub platform has much to offer on that front and `orchestrator` now uses GitHub Actions more heavily for testing.

GitHub Actions provide a way to run code in a container in the context of the repository. The most common use case is to run CI tests on receiving a Pull Request. Indeed, when GitHub Actions became available, we switched out of Travis CI and into Actions for `orchestrator`‘s CI.

Today, `orchestrator` runs three different tests:

- Build, unit testing, integration testing, code & doc validation
- Upgrade testing
- System testing

To highlight what each does: Continue reading » “orchestrator: what’s new in CI, testing & development”

## The state of Orchestrator, 2020 (spoiler: healthy)

MySQL GitHub MySQL Open Source orchestrator February 18, 2020February 18, 2020

This post serves as a pointer to my previous announcement about The state of Orchestrator, 2020.

Thank you to Tom Krouper who applied his operational engineer expertise to content publishing problems.

## The state of Orchestrator, 2020 (spoiler: healthy)

MySQL GitHub Open Source orchestrator February 18, 2020

Yesterday was my last day at GitHub, and this post explains what this means for `orchestrator`. First, a quick historical review:

- 2014: I began work on `orchestrator` at Outbrain, as https://github.com/outbrain/orchestrator. I authored several open source projects while working for Outbrain, and created `orchestrator` to solve discovery, visualization and simple refactoring needs. Outbrain was happy to have the project developed as a public, open source repo from day 1, and it was released under the Apache 2 license. Interestingly, the idea to develop `orchestrator` came after I attended Percona Live Santa Clara 2014 and watched “ChatOps: How GitHub Manages MySQL” by one Sam Lambert.
- 2015: Joined Booking.com where my main focus was to redesign and solve issues with the existing high availability setup. With Booking.com’s support, I continued work on `orchestrator`, pursuing better failure detection and recovery processes. Booking.com was an incredible playground and testbed for `orchestrator`, a massive deployment of multiple MySQL/MariaDB flavors and configuration.
- 2016 – 2020: Joined GitHub. GitHub adopted `orchestrator` and I developed it under GitHub’s own org, at https://github.com/github/orchestrator. It became a core component in github.com’s high availability design, running failure detection and recoveries across sites and geographical regions, with more to come. These 4+ years have been critical to `orchestrator`‘s development and saw its widespread use. At this time I’m aware of multiple large-scale organizations using `orchestrator` for high availability and failovers. Some of these are GitHub, Booking.com, Shopify, Slack, Wix, Outbrain, and more. `orchestrator` is the underlying failover mechanism for vitess, and is also included in Percona’s PMM. These years saw a significant increase in community adoption and contributions, in published content, such as Pythian and Percona technical blog posts, and, not surprisingly, increase in issues and feature requests.

###
2020

GitHub was very kind to support moving the `orchestrator` repo under my own https://github.com/openark org. This means all issues, pull requests, releases, forks, stars and watchers have automatically transferred to the new location: https://github.com/openark/orchestrator. The old links do a “follow me” and implicitly direct to the new location. All external links to code and docs still work. I’m grateful to GitHub for supporting this transfer.

I’d like to thank all the above companies for their support of `orchestrator` and of open source in general. Being able to work on the same product throughout three different companies is mind blowing and an incredible opportunity. `orchestrator` of course remains open source and licensed with Apache 2. Existing Copyrights are unchanged.

As for what’s next: some personal time off, please understand if there’s delays to reviews/answers. My intention is to continue developing `orchestrator`. Naturally, the shape of future development depends on how `orchestrator` meets my future work. Nothing changes in that respect: my focus on `orchestrator` has always been first and foremost the pressing business needs, and then community support as possible. There are some interesting ideas by prominent `orchestrator` users and adopters and I’ll share more thoughts in due time.

## Un-split brain MySQL via gh-mysql-rewind

MySQL Open Source Replication March 5, 2019March 5, 2019

We are pleased to release gh-mysql-rewind, a tool that allows us to move MySQL back in time, automatically identify and rewind split brain changes, restoring a split brain server into a healthy replication chain.

I recently had the pleasure of presenting `gh-mysql-rewind` at FOSDEM. Video and slides are available. Consider following along with the video.

## Motivation

Consider a split brain scenario: a “standard” MySQL replication topology suffered network isolation, and one of the replicas was promoted as new master. Meanwhile, the old master was still receiving writes from co-located apps.

Once the network isolation is over, we have a new master and an old master, and a split-brain situation: some writes only took place on one master; others only took place on the other. What if we wanted to converge the two? What paths do we have to, say, restore the old, demoted master, as a replica of the newly promoted master?

The old master is unlikely to agree to replicate from the new master. Changes have been made. `AUTO_INCREMENT` values have been taken. `UNIQUE` constraints will fail.

A few months ago, we at GitHub had exactly this scenario. An entire data center went network isolated. Automation failed over to a 2nd DC. Masters in the isolated DC meanwhile kept receiving writes. At the end of the failover we ended up with a split brain scenario – which we expected. However, an additional, unexpected constraint forced us to fail back to the original DC.

We had to make a choice: we’ve already operated for a long time in the 2nd DC and took many writes, that we were unwilling to lose. We were OK to lose (after auditing) the few seconds of writes on the isolated DC. But, how do we converge the data?

Backups are the trivial way out, but they incur long recovery time. Shipping backup data over the network for dozens of servers takes time. Restore time, catching up with changes since backup took place, warming up the servers so that they can handle production traffic, all take time.

Could we have reduces the time for recovery?

Continue reading » “Un-split brain MySQL via gh-mysql-rewind”

## Using dbdeployer in CI tests

MySQL dbdeployer gh-ost Open Source Replication testing February 20, 2018May 11, 2020

I was very pleased when Giuseppe Maxia (aka datacharmer) unveiled dbdeployer in his talk at pre-FOSDEM MySQL day. The announcement came just at the right time. I wish to briefly describe how we use `dbdeployer` (work in progress).

### The case for gh-ost

A user opened an issue on `gh-ost`, and the user was using MySQL `5.5`. `gh-ost` is being tested on `5.7` where the problem does not reproduce. A discussion with Gillian Gunson raised the concern of not testing on all versions. Can we run `gh-ost` tests for all MySQL/Percona/MariaDB versions? Should we? How easy would it be?

### gh-ost tests

`gh-ost` has three different test types:

- Unit tests: these are plain `golang` logic tests which are very easy and quick to run.
- Integration tests: the topic of this post, see following. Today these do not run as part of an automated CI testing.
- System tests: putting our production tables to the test, continuously migrating our production data on dedicated replicas, verifying checksums are identical and data is intact, read more.

Unit tests are already running as part of automated CI (every PR is subjected to those tests). Systems tests are clearly tied to our production servers. What’s the deal with the integration tests? Continue reading » “Using dbdeployer in CI tests”

## orchestrator 3.0.6: faster crash detection & recoveries, auto Pseudo-GTID, semi-sync and more

MySQL High availability Open Source orchestrator Replication January 29, 2018January 29, 2018

`orchestrator` 3.0.6 is released and includes some exciting improvements and features. It quickly follows up on 3.0.5 released recently, and this post gives a breakdown of some notable changes:

### Faster failure detection

Recall that `orchestrator` uses a holistic approach for failure detection: it reads state not only from the failed server (e.g. master) but also from its replicas. `orchestrator` now detects failure faster than before:

- A detection cycle has been eliminated, leading to quicker resolution of a failure. On our setup, where we poll servers every `5sec`, failure detection time dropped from `7-10sec` to `3-5sec`, keeping reliability. The reduction in time does not lead to increased false positives.
  Side note: you may see increased not-quite-failure analysis such as “I can’t see the master” (`UnreachableMaster`).
- Better handling of network scenarios where packets are dropped. Instead of hanging till TCP timeout, `orchestrator` now observes server discovery asynchronously. We have specialized failover tests that simulate dropped packets. The change reduces detection time by some `5sec`.

### Faster master recoveries

Promoting a new master is a complex task which attempts to promote the best replica out of the pool of replicas. It’s not always the most up-to-date replica. The choice varies depending on replica configuration, version, and state.

With recent changes, `orchestrator` is able to to recognize, early on, that the replica it would like to promote as master is ideal. Assuming that is the case, `orchestrator` is able to immediate promote it (i.e. run hooks, set `read_only=0` etc.), and run the rest of the failover logic, i.e. the rewiring of replicas under the newly promoted master, asynchronously.

This allows the promoted server to take writes sooner, even while its replicas are not yet connected. It also means external hooks are executed sooner.

Between faster detection and faster recoveries, we’re looking at some `10sec` reduction in overall recovery time: from moment of crash to moment where a new master accepts writes. We stand now at `< 20sec` in almost all cases, and `< 15s` in optimal cases. Those times are measured on our failover tests.

We are working on reducing failover time unrelated to `orchestrator` and hope to update soon.

### Automated Pseudo-GTID

As reminder, Pseudo-GTID is an alternative to GTID, without the kind of commitment you make with GTID. It provides similar “point your replica under any other server” behavior GTID allows. Continue reading » “orchestrator 3.0.6: faster crash detection & recoveries, auto Pseudo-GTID, semi-sync and more”

## gh-ost 1.0.42 released: JSON support, optimizations

MySQL gh-ost Open Source September 14, 2017September 15, 2017

gh-ost 1.0.42 is released and available for download.

### JSON

MySQL 5.7’s `JSON` data type is now supported.

There is a soft-limitation, that your `JSON` may not be part of your `PRIMARY KEY`. Currently this isn’t even supported by MySQL anyhow.

### Performance

Two noteworthy changes are:

- Client side prepared statements reduce network traffic and round trips to the server.
- Range query iteration avoids creating temporary tables and filesorting.

We’re not running benchmarks at this time to observe performance gains.

### 5.7

More tests validating `5.7` compatibility (at this time GitHub runs MySQL `5.7` in production).

### Ongoing

Many other changes included.

We are grateful for all community feedback in form of open Issues, Pull Requests and questions!

`gh-ost` is authored by GitHub. It is free and open source and is available under the MIT license.

### Speaking

In two weeks time, Jonah Berquist will present gh-ost: Triggerless, Painless, Trusted Online Schema Migrations at Percona Live, Dublin.

Tom Krouper and myself will present MySQL Infrastructure Testing Automation at GitHub, where, among other things, we describe how we test `gh-ost` in production.

## Speaking at Percona Live Dublin: keynote, orchestrator tutorial, MySQL testing automation

MySQL Open Source PerconaLive secondary Speaking September 13, 2017September 13, 2017

I’m looking forward to a busy Percona Live Dublin conference, delivering three talks. Chronologically, these are:

- Practical orchestrator tutorial
  Attend this 3 hour tutorial for a thorough overview on orchestrator: what, why, how to configure, best advice, deployments, failovers, security, high availability, common operations, …
  We will of course discuss the new orchestrator/raft setup and share our experience running it in production.
  The tutorial will allow for general questions from the audience and open discussions.
- Why Open Sourcing Our Database Tooling was the Smart Decision
  What it says. A 10 minute journey advocating for open sourcing infrastructure.
- MySQL Infrastructure Testing Automation at GitHub
  Co-presenting with Tom Krouper, we share how & why we run infrastructure tests in and near production that gives us trust in many of our ongoing, ever changing operations. Essentially this is “why you should feel OK trusting us with your data”.

See you there!

## orchestrator 3.0.2 GA released: raft consensus, SQLite

MySQL Open Source orchestrator September 12, 2017September 12, 2017

orchestrator 3.0.2 GA is released and available for download (see also packagecloud repository).

`3.0.2` is the first stable release in the `3.0*` series, introducing (recap from 3.0 pre-release announcement):

### orchestrator/raft

Raft is a consensus protocol, supporting leader election and consensus across a distributed system. In an `orchestrator/raft` setup `orchestrator` nodes talk to each other via raft protocol, form consensus and elect a leader. Each `orchestrator` node has its own dedicated backend database. The backend databases do not speak to each other; only the `orchestrator` nodes speak to each other.

No MySQL replication setup needed; the backend DBs act as standalone servers. In fact, the backend server doesn’t have to be MySQL, and `SQLite`is supported. `orchestrator` now ships with `SQLite` embedded, no external dependency needed.

For details, please refer to the documentation:

- orchestrator/raft: overview
- orchestrator/raft vs. shared backend DB setup, comparison

### SQLite

Suggested and requested by many, is to remove `orchestrator`‘s own dependency on a MySQL backend. `orchestrator` now supports a SQLite backend.

`SQLite` is a transactional, relational, embedded database, and as of `3.0` it is embedded within `orchestrator`, no external dependency required.

### orchestrator-client

orchestrator-client is a client shell script which mimics the command line interface, while running `curl | jq` requests against the HTTP API. It stands to simplify your deployments: interacting with the orchestrator service via `orchestrator-client` is easier and only requires you to place a shell script (this is as opposed to installing the `orchestrator` binary + configuration file).

`orchestrator-client` is the way to interact with your `orchestrator/raft` cluster. `orchestrator-client` now has its own `RPM/deb` release package.

You may still use the web interface, web API ; and a special `--ignore-raft-setup` keeps power at your hand (use at your own risk).

### State of orchestrator/raft

`orchestrator/raft` is a big change: Continue reading » “orchestrator 3.0.2 GA released: raft consensus, SQLite”