This is the first in a series of posts reviewing methods for MySQL master discovery: the means by which an application connects to the master of a replication tree. Moreover, the means by which, upon master failover, it identifies and connects to the newly promoted master.<\/p>\n
These posts are not concerned with the manner by which the replication failure detection and recovery take place. I will share We discuss asynchronous (or semi-synchronous) replication, a classic single-master-multiple-replicas setup. A later post will briefly discuss synchronous replication (Galera\/XtraDB Cluster\/InnoDB Cluster).<\/p>\n In DNS master discovery applications connect to the master via a name that gets resolved to the master’s box. By way of example, apps would target the masters of different clusters by connecting to <\/p>\n Issues for concern are:<\/p>\n As long as things are stable and going well, discovery via DNS makes sense. Trouble begins when the master fails over. Assume Our failover solution has promoted Master Say During that time they will continue to attempt connecting to Master Again, assume Clients who will require between We wish to replace the master, for maintenance reasons. We successfully and gracefully promote And still our clients take The above numbers are just illustrative. Perhaps DNS deployment is quicker than For planned takeover we can first deploy a change to the You may choose to restart apps upon DNS deployment. This emulates apps’ awareness of the change.<\/p>\n In the above:<\/p>\n See orchestrator configuration<\/a> documentation.<\/p>\n This is the first in a series of posts reviewing methods for MySQL master discovery: the means by which an application connects to the master of a replication tree. Moreover, the means by which, upon master failover, it identifies and connects to the newly promoted master. These posts are not concerned with the manner by […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"enabled":false},"version":2}},"categories":[5],"tags":[62,108,8],"class_list":["post-7861","post","type-post","status-publish","format-standard","hentry","category-mysql","tag-high-availability","tag-orchestrator","tag-replication"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p2bZZp-22N","_links":{"self":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/7861","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/comments?post=7861"}],"version-history":[{"count":8,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/7861\/revisions"}],"predecessor-version":[{"id":7908,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/7861\/revisions\/7908"}],"wp:attachment":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/media?parent=7861"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/categories?post=7861"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/tags?post=7861"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}orchestrator<\/code> specific configuration\/advice, and point out where cross DC orchestrator\/raft<\/code> setup plays part in discovery itself, but for the most part any recovery tool such as MHA<\/code>, replication-manager<\/code>, severalnines<\/code> or other, is applicable.<\/p>\nMaster discovery via DNS<\/h3>\n
cluster1-writer.example.net<\/code>, cluster2-writer.example.net<\/code>, etc. It is up for the DNS to resolve those names to IPs.<\/p>\n\n
TTL<\/code> (Time To Live) such that clients can cache the IP associated with a name for a given number of seconds. What is that TTL<\/code>?<\/li>\n<\/ul>\nM<\/code> used to be the master, but got demoted. Assume R<\/code> used to be a replica, that got promoted and is now effectively the master of the topology.<\/p>\nR<\/code>, and now needs to somehow apply the change, such that the apps connect to R<\/code> instead of M<\/code>. Some notes:<\/p>\n\n
cluster1-writer.example.net<\/code>, cluster2-writer.example.net<\/code>, etc.<\/li>\nTTL<\/code>.<\/li>\n<\/ul>\nA non planned failover illustration #1<\/h3>\n
M<\/code> dies. R<\/code> gets promoted. Our tool instructs all DNS servers on all DCs to update the IP address.<\/p>\nTTL<\/code> is 60<\/code> seconds. Say update to all DNS servers takes 10<\/code> seconds. We will have between 10<\/code> and 70<\/code> seconds until all clients connect to the new master R<\/code>.<\/p>\nM<\/code>. Since M<\/code> is dead, those attempts will fail (thankfully).<\/p>\nA non planned failover illustration #2<\/h3>\n
M<\/code> gets network isolated for 30<\/code> seconds, during which time we failover. R<\/code> gets promoted. Our tool instructs all DNS servers on all DCs to update the IP address.<\/p>\nTTL<\/code> is 60<\/code> seconds. As before, it will take between 10<\/code> and 70<\/code> seconds for clients to learn of the new IP.<\/p>\n40<\/code> and 70<\/code> seconds to learn of the new IP will, however, hit an unfortunate scenario: the old master M<\/code> reappears on the grid. Those clients will successfully reconnect to M<\/code> and issue writes, leading to data loss (writes to M<\/code> no longer replicate anywhere).<\/p>\nPlanned failover illustration<\/h3>\n
R<\/code>. We need to change DNS records. Since this is a planned failover, we set the old master to read_only=1<\/code>, or even better, we network isolated it.<\/p>\n10<\/code> to 70<\/code> seconds to recognize the new master.<\/p>\nDiscussion<\/h3>\n
10<\/code> seconds. You should do your own math.<\/p>\nTTL<\/code> is a compromise which you can tune. Setting lower TTL<\/code> will mitigate the problem, but will cause more hits on the DNS servers.<\/p>\nTTL<\/code>, to, say, 2sec<\/code>, wait 60sec<\/code>, then deploy the IP change, then restore TTL<\/code> to 60<\/code>.<\/p>\nSample orchestrator configuration<\/h3>\n
orchestrator<\/code> configuration would look like this:<\/p>\n \"ApplyMySQLPromotionAfterMasterFailover\": true,\n \"PostMasterFailoverProcesses\": [\n \"\/do\/what\/you\/gotta\/do to apply dns change for {failureClusterAlias}-writer.example.net to {successorHost}\"\n ], \n<\/code><\/pre>\n\n
ApplyMySQLPromotionAfterMasterFailover<\/code> instructs orchestrator<\/code> to set read_only=0; reset slave all<\/code> on promoted server.<\/li>\nPostMasterFailoverProcesses<\/code> really depends on your setup. But orchestrator<\/code> will supply with hints to your scripts: identity of cluster, identity of successor.<\/li>\n<\/ul>\nAll posts in this series<\/h3>\n
\n