{"id":6867,"date":"2014-06-09T09:02:26","date_gmt":"2014-06-09T07:02:26","guid":{"rendered":"http:\/\/code.openark.org\/blog\/?p=6867"},"modified":"2014-06-09T09:02:26","modified_gmt":"2014-06-09T07:02:26","slug":"introducing-orchestrator-manage-and-visualize-your-mysql-replication-topologies-and-get-home-for-dinner","status":"publish","type":"post","link":"https:\/\/code.openark.org\/blog\/mysql\/introducing-orchestrator-manage-and-visualize-your-mysql-replication-topologies-and-get-home-for-dinner","title":{"rendered":"Introducing Orchestrator: manage and visualize your MySQL replication topologies and get home for dinner"},"content":{"rendered":"<p>I&#8217;m happy to announce the availability of <a href=\"http:\/\/www.outbrain.com\/\">Outbrain<\/a>&#8216;s <strong><a href=\"https:\/\/github.com\/outbrain\/orchestrator\">Orchestrator<\/a><\/strong>: MySQL replication management &amp; visualization tool.<\/p>\n<blockquote><p><a href=\"http:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/05\/orchestrator-simple.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-6898\" src=\"http:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/05\/orchestrator-simple.png\" alt=\"orchestrator - simple topology\" width=\"785\" height=\"353\" srcset=\"https:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/05\/orchestrator-simple.png 785w, https:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/05\/orchestrator-simple-300x134.png 300w\" sizes=\"auto, (max-width: 785px) 100vw, 785px\" \/><\/a><\/p><\/blockquote>\n<ul>\n<li>Orchestrator reads your replication topologies (give it one server &#8211; be it master or slave &#8211; in each topology, and it will reveal the rest).<\/li>\n<li>It keeps a state of this topology.<\/li>\n<li>It can continuously poll your servers to get an up to date topology map.<\/li>\n<li>It visualizes the topology in a clear and slick D3 tree.<\/li>\n<li>It allows you to <em>modify your topology;<\/em> move slaves around. You can use the command line variation, the JSON API, or you can use the web interface.<\/li>\n<\/ul>\n<p>Quick links:\u00a0<strong><a href=\"https:\/\/github.com\/outbrain\/orchestrator\/wiki\/Orchestrator-Manual\">Orchestrator Manual<\/a><\/strong>, <strong><a href=\"https:\/\/github.com\/outbrain\/orchestrator\/wiki\/FAQ\">FAQ<\/a><\/strong>, <strong><a href=\"https:\/\/github.com\/outbrain\/orchestrator\/releases\">Downloads<\/a><\/strong><\/p>\n<h4>Nothing like nice screenshots<\/h4>\n<p>To move slaves around the topology (repoint a slave to a different master) through <em>orchestrator<\/em>&#8216;s web interface, we use <em>Drag and Drop<\/em>,<!--more--><\/p>\n<p>just<\/p>\n<blockquote><p><a href=\"http:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/05\/orchestrator-simple-drag.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-6899\" src=\"http:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/05\/orchestrator-simple-drag.png\" alt=\"orchestrator - simple topology - rag\" width=\"785\" height=\"353\" srcset=\"https:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/05\/orchestrator-simple-drag.png 785w, https:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/05\/orchestrator-simple-drag-300x134.png 300w\" sizes=\"auto, (max-width: 785px) 100vw, 785px\" \/><\/a><\/p><\/blockquote>\n<p>like<\/p>\n<blockquote><p><a href=\"http:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-drag-hover.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-6900\" src=\"http:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-drag-hover.png\" alt=\"orchestrator - simple topology -drag-hover\" width=\"785\" height=\"353\" srcset=\"https:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-drag-hover.png 785w, https:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-drag-hover-300x134.png 300w\" sizes=\"auto, (max-width: 785px) 100vw, 785px\" \/><\/a><\/p><\/blockquote>\n<p>that.<\/p>\n<blockquote><p><a href=\"http:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-dropped.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-6902\" src=\"http:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-dropped.png\" alt=\"orchestrator - simple topology - dropped\" width=\"870\" height=\"272\" srcset=\"https:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-dropped.png 870w, https:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-dropped-300x93.png 300w\" sizes=\"auto, (max-width: 870px) 100vw, 870px\" \/><\/a><\/p><\/blockquote>\n<h4>Safety<\/h4>\n<p>Orchestrator keeps you safe. It does so by:<\/p>\n<ul>\n<li>Correctly calculating the binary log files &amp; positions (aka coordinates) of the slave you&#8217;re moving, its current master, its new master; it properly stops, starts and stalls your replication till everything is in sync.<\/li>\n<li>Helping you to avoid shooting yourself in the leg. It will not allow moving a slave that uses <strong>STATEMENT<\/strong> based replication under a <strong>ROW<\/strong> based replication server. Or a <strong>5.5<\/strong> under a <strong>5.6<\/strong>. Or anything under a server that doesn&#8217;t have binary logs. Or <strong>log_slave_updates<\/strong>. Or if one of the servers involed lags too much. Or more&#8230;<\/li>\n<\/ul>\n<h4>Visualization<\/h4>\n<p>It also points out a few problems, visually. While it is <em>not<\/em> &#8211; and will not be &#8211; a monitoring tool, it requires some replication status info for its own purposes. Too much lag? Replication not working? Server cannot be accessed? Server under maintenance? This all shows up in your topology. We use it a lot to get a holistic view over our <em>current<\/em> replication topologies status.<\/p>\n<blockquote><p><a href=\"http:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-with-problems.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-6932\" src=\"http:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-with-problems.png\" alt=\"orchestrator-simple-with-problems\" width=\"696\" height=\"353\" srcset=\"https:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-with-problems.png 696w, https:\/\/code.openark.org\/blog\/wp-content\/uploads\/2014\/06\/orchestrator-simple-with-problems-300x152.png 300w\" sizes=\"auto, (max-width: 696px) 100vw, 696px\" \/><\/a><\/p><\/blockquote>\n<h4>State<\/h4>\n<p><em>Orchestrator<\/em> keeps the state of your topologies. Unlike other tools that will drill down from the master and just pick up on whatever&#8217;s connected <em>right now<\/em>, <em>orchestrator<\/em> remembers what used to be connected, too. If a slave is not replicating at this very moment, that does not mean it&#8217;s not part of the topology. Same for a MySQL service that has been temporarily stopped. And this includes all their slaves, if any. Until told otherwise (or until too much time passes and a server is assumed dead), <em>orchestrator<\/em> keeps the map intact.<\/p>\n<h4>Maintenance<\/h4>\n<p><em>Orchestrator<\/em> supports a maintenance-mode state; it&#8217;s a flag saying &#8220;this server is in maintenance mode right now&#8221;. But this flag includes an owner and a reason for audit purposes. And while a server is under maintenance, orchestrator will disallow replication topology changes that include this server.<\/p>\n<h4>Audit<\/h4>\n<p>Operations performed via orchestrator are audited (well, almost all). You have a complete history on what slave has been moved from where to where; what server has been taken under maintenance and when, etc.<\/p>\n<h4>Automation<\/h4>\n<p>The most important thing is of course automating error-prone human sequences of actions. Repointing slaves is a mess (when you don&#8217;t have GTIDs). Automation saves time and greatly reduces the possibility that something goes wrong (of course never eliminates). We happen to use\u00a0<em>orchestrator<\/em> at Outbrain on production, and twice in the past month had major events where <em>orchestrator<\/em> saved us many hours and worry.<\/p>\n<h4>Support<\/h4>\n<p><em>Orchestrator<\/em> supports &#8220;standard&#8221; replication: log file:pos kind of replication. Non GTID, non-parallel. Good (?) old replication.<\/p>\n<p>To quickly explain why not GTID: we&#8217;re using MySQL <strong>5.5<\/strong>. We&#8217;ve had <a href=\"http:\/\/code.openark.org\/blog\/mysql\/the-mystery-of-mysql-5-6-excessive-buffer-pool-flushing\">issues<\/a> while evaluating <strong>5.6<\/strong>; and besides, migrating to GTID is <a href=\"http:\/\/blog.booking.com\/mysql-5.6-gtids-evaluation-and-online-migration.html\">a mess<\/a> (several solutions or proposed solutions seem to exist). On April&#8217;s Percona Live event, I realized only a few companies actually use MySQL <strong>5.6<\/strong>, and a few of these use GTID. We&#8217;re not doing GTID (yet), as much as we would want to. Not there yet.<\/p>\n<p>Read the <strong><a href=\"https:\/\/github.com\/outbrain\/orchestrator\/wiki\/FAQ\">FAQ<\/a><\/strong> for further questions on supported replication technologies.<\/p>\n<h4>How do you like it?<\/h4>\n<p><em>Orchestrator<\/em> can run as a command line tool (no need for Web). It can server HTTP JSON API (no need for visualization) or it can server as HTTP web interface (no need to use command line options). Have it your way.<\/p>\n<h4>The technology stack<\/h4>\n<p><em>Orchestrator<\/em> is written in <a href=\"http:\/\/golang.org\/\">Go<\/a>, with <a href=\"https:\/\/github.com\/go-martini\/martini\">Martini<\/a> as web framework; <a href=\"http:\/\/dev.mysql.com\/\">MySQL<\/a> as backend database; <a href=\"http:\/\/d3js.org\/\">D3<\/a>, <a href=\"http:\/\/jquery.com\/\">jQuery<\/a> &amp; <a href=\"http:\/\/getbootstrap.com\/\">bootstrap<\/a> for frontend.<\/p>\n<h4>License<\/h4>\n<p><em>Orchestrator<\/em> is released as open source under the <a href=\"https:\/\/raw.githubusercontent.com\/outbrain\/orchestrator\/master\/LICENSE\">Apache 2.0 license<\/a> and is available at: <a href=\"https:\/\/github.com\/outbrain\/orchestrator\">https:\/\/github.com\/outbrain\/orchestrator<\/a><\/p>\n<h4>Documentation<\/h4>\n<p>Read the <strong><a href=\"https:\/\/github.com\/outbrain\/orchestrator\/wiki\/Orchestrator-Manual\">Manual<\/a><\/strong><\/p>\n<h4>Download<\/h4>\n<p>Get the <strong><a href=\"https:\/\/github.com\/outbrain\/orchestrator\/releases\">bundled binary+web files<\/a><\/strong> tarball, RPM or DEB packages. Or just clone the project. It&#8217;s free.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;m happy to announce the availability of Outbrain&#8216;s Orchestrator: MySQL replication management &amp; visualization tool. Orchestrator reads your replication topologies (give it one server &#8211; be it master or slave &#8211; in each topology, and it will reveal the rest). It keeps a state of this topology. It can continuously poll your servers to get [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"enabled":false},"version":2}},"categories":[5],"tags":[57,108,8,100],"class_list":["post-6867","post","type-post","status-publish","format-standard","hentry","category-mysql","tag-open-source","tag-orchestrator","tag-replication","tag-tools"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p2bZZp-1ML","_links":{"self":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/6867","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/comments?post=6867"}],"version-history":[{"count":29,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/6867\/revisions"}],"predecessor-version":[{"id":6936,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/6867\/revisions\/6936"}],"wp:attachment":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/media?parent=6867"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/categories?post=6867"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/tags?post=6867"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}