Skip to content

# MySQL master discovery methods, part 5: Service discovery & Proxy

MySQL High availability orchestrator Replication May 14, 2018May 22, 2018

This is the fifth in a series of posts reviewing methods for MySQL master discovery: the means by which an application connects to the master of a replication tree. Moreover, the means by which, upon master failover, it identifies and connects to the newly promoted master.

These posts are not concerned with the manner by which the replication failure detection and recovery take place. I will share `orchestrator` specific configuration/advice, and point out where cross DC `orchestrator/raft` setup plays part in discovery itself, but for the most part any recovery tool such as `MHA`, `replication-manager`, `severalnines` or other, is applicable.

We discuss asynchronous (or semi-synchronous) replication, a classic single-master-multiple-replicas setup. A later post will briefly discuss synchronous replication (Galera/XtraDB Cluster/InnoDB Cluster).

### Master discovery via Service discovery and Proxy

Part 4 presented with an anti-pattern setup, where a proxy would infer the identify of the master by drawing conclusions from backend server checks. This led to split brains and undesired scenarios. The problem was the loss of context.

We re-introduce a service discovery component (illustrated in part 3), such that:

- The app does not own the discovery, and
- The proxy behaves in an expected and consistent way.

In a failover/service discovery/proxy setup, there is clear ownership of duties:

- The failover tool own the failover itself and the master identity change notification.
- The service discovery component is the source of truth as for the identity of the master of a cluster.
- The proxy routes traffic but does not make routing decisions.
- The app only ever connects to a single target, but should allow for a brief outage while failover takes place.

Depending on the technologies used, we can further achieve:

- Hard cut for connections to old, demoted master `M`.
- Black/hold off for incoming queries for the duration of failover.

We explain the setup using the following assumptions and scenarios:

- All clients connect to master via `cluster1-writer.example.net`, which resolves to a proxy box.
- We fail over from master `M` to promoted replica `R`.

### A non planned failover illustration #1

Master `M` has died, the box had a power failure. `R` gets promoted in its place. Our recovery tool:

- Updates service discovery component that `R` is the new master for `cluster1`.

The proxy:

- Either actively or passively learns that `R` is the new master, rewires all writes to go to `R`.
- If possible, kills existing connections to `M`.

The app:

- Needs to know nothing. Its connections to `M` fail, it reconnects and gets through to `R`.

### A non planned failover illustration #2

Master `M` gets network isolated for `10` seconds, during which time we failover. `R` gets promoted.

Everything is as before.

If the proxy kills existing connections to `M`, then the fact `M` is back alive turns meaningless. No one gets through to `M`. Clients were never aware of its identity anyhow, just as they are unaware of `R`‘s identity.

### Planned failover illustration

We wish to replace the master, for maintenance reasons. We successfully and gracefully promote `R`.

- In the process of promotion, `M` turned read-only.
- Immediately following promotion, our failover tool updates service discovery.
- Proxy reloads having seen the changes in service discovery.
- Our app connects to `R`.

### Discussion

This is a setup we use at GitHub in production. Our components are:

- `orchestrator` for failover tool.
- Consul for service discovery.
- GLB (HAProxy) for proxy
- Consul template running on proxy hosts:
  - listening on changes to Consul’s KV data
  - Regenerate `haproxy.cfg` configuration file
  - `reload` haproxy

As mentioned earlier, the apps need not change anything. They connect to a name that is always resolved to proxy boxes. There is never a DNS change.

At the time of failover, the service discovery component must be up and available, to catch the change. Otherwise we do not strictly require it to be up at all times.

For high availability we will have multiple proxies. Each of whom must listen on changes to K/V. Ideally the name (`cluster1-writer.example.net` in our example) resolves to any available proxy box.

- This, in itself, is a high availability issue. Thankfully, managing the HA of a proxy layer is simpler than that of a MySQL layer. Proxy servers tend to be stateless and equal to each other.
- See GLB as one example for a highly available proxy layer. Cloud providers, Kubernetes, two level layered proxies, Linux Heartbeat, are all methods to similarly achieve HA.

See also:

- MySQL High Availability With HAProxy, Consul And Orchestrator
- Automatic Failovers with Kubernetes using Orchestrator, ProxySQL and Zookeeper
- Orchestrating ProxySQL with Orchestrator and Consul

### Sample orchestrator configuration

An `orchestrator` configuration would look like this:

      "ApplyMySQLPromotionAfterMasterFailover": true,
      "KVClusterMasterPrefix": "mysql/master",
      "ConsulAddress": "127.0.0.1:8500",
      "ZkAddress": "srv-a,srv-b:12181,srv-c",
      "PostMasterFailoverProcesses": [
        “/just/let/me/know about failover on {failureCluster}“,
      ],

In the above:

- If `ConsulAddress` is specified, `orchestrator` will update given Consul setup with K/V changes.
- At `3.0.10`, ZooKeeper, via `ZkAddress`, is still not supported by `orchestrator`.
- `PostMasterFailoverProcesses` is here just to point out hooks are not strictly required for the operation to run.

See orchestrator configuration documentation.

### All posts in this series

- MySQL master discovery methods, part 1: DNS
- MySQL master discovery methods, part 2: VIP & DNS
- MySQL master discovery methods, part 3: app & service discovery
- MySQL master discovery methods, part 4: Proxy heuristics
- MySQL master discovery methods, part 5: Service discovery & Proxy
- MySQL master discovery methods, part 6: other methods

- Twitter
- LinkedIn
- WhatsApp
- Email
- Facebook
- Print

### Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe
Notify of
wpDiscuz
Send to Email Address Your Name Your Email Address Cancel
Post was not sent - check your email addresses!
Email check failed, please try again
Sorry, your blog cannot share posts by email.