{"id":8077,"date":"2020-05-11T10:01:08","date_gmt":"2020-05-11T08:01:08","guid":{"rendered":"http:\/\/code.openark.org\/blog\/?p=8077"},"modified":"2020-05-12T09:13:37","modified_gmt":"2020-05-12T07:13:37","slug":"orchestrator-whats-new-in-ci-testing-development","status":"publish","type":"post","link":"https:\/\/code.openark.org\/blog\/mysql\/orchestrator-whats-new-in-ci-testing-development","title":{"rendered":"orchestrator: what&#8217;s new in CI, testing &#038; development"},"content":{"rendered":"<p>Recent focus on development &amp; testing yielded with new <a href=\"https:\/\/github.com\/openark\/orchestrator\">orchestrator<\/a> environments and offerings for developers and with increased reliability and trust. This post illustrates the new changes, and see <a href=\"https:\/\/github.com\/openark\/orchestrator\/tree\/master\/docs#developers\">Developers<\/a> section on the official documentation for more details.<\/p>\n<h2>Testing<\/h2>\n<p>In the past four years <code>orchestrator<\/code> was <a href=\"https:\/\/github.blog\/2016-12-08-orchestrator-github\/\">developed at GitHub<\/a>, and using GitHub&#8217;s environments for <a href=\"https:\/\/github.blog\/2017-07-06-mysql-testing-automation-at-github\/\">testing<\/a>. This is very useful for testing <code>orchestrator<\/code>&#8216;s behavior within GitHub, interacting with its internal infrastructure, and validating failover behavior in a production environment. These tests and their results are not visible to the public, though.<\/p>\n<p>Now that <code>orchestrator<\/code> is developed <a href=\"http:\/\/code.openark.org\/blog\/mysql\/the-state-of-orchestrator-2020-spoiler-healthy\">outside GitHub<\/a> (that is, outside GitHub the <em>company<\/em>, not GitHub the <em>platform<\/em>) I wanted to improve on the testing framework, making it visible, accessible and contribute-able to the community. Thankfully, the GitHub platform has much to offer on that front and <code>orchestrator<\/code> now uses <a href=\"https:\/\/github.com\/features\/actions\">GitHub Actions<\/a> more heavily for testing.<\/p>\n<p>GitHub Actions provide a way to run code in a container in the context of the repository. The most common use case is to run CI tests on receiving a Pull Request. Indeed, when GitHub Actions became available, we switched out of Travis CI and into Actions for <code>orchestrator<\/code>&#8216;s CI.<\/p>\n<p>Today, <code>orchestrator<\/code> runs three different tests:<\/p>\n<ul>\n<li>Build, unit testing, integration testing, code &amp; doc validation<\/li>\n<li>Upgrade testing<\/li>\n<li>System testing<\/li>\n<\/ul>\n<p>To highlight what each does:<!--more--><\/p>\n<h3>Build, unit testing, integration testing<\/h3>\n<p>Based on the original CI (and possibly will split into distinct tests), this CI Action compiles the code, runs unit tests, runs the suite of <a href=\"https:\/\/github.com\/openark\/orchestrator\/tree\/master\/tests\/integration\">integration tests<\/a> (spins up both <code>MySQL<\/code> and <code>SQLite<\/code> databases and runs a series of tests on each backend), this CI job is the &#8220;basic&#8221; test to see that the contributed code even makes sense.<\/p>\n<p>What&#8217;s new in this test is that it now produces an <em>artifact<\/em>: an <code>orchestrator<\/code> binary for Linux\/amd64. This is again a feature for GitHub Actions; the artifact is kept for a couple months or so per Actions retention policy. <a href=\"https:\/\/github.com\/openark\/orchestrator\/actions\/runs\/94337568\">Here<\/a>&#8216;s an example; by the time you read this the binary artifact may or may not still be there.<\/p>\n<p>This means you don&#8217;t actually need a development environment on your laptop to be able to build and <code>orchestrator<\/code> binary. More on this later.<\/p>\n<h3>Upgrade testing<\/h3>\n<p>Until recently not formalized; I&#8217;d test upgrades by deploying them internally at GitHub onto a staging environment. Now upgrades are tested per Pull Request: we spin up a container, deploy <code>orchestrator<\/code> from <code>master<\/code> branch using both <code>MySQL<\/code> and <code>SQLite<\/code> backends, then checkout the PR branch, and redeploy <code>orchestrator<\/code> using the existing backends &#8212; this verifies that at least backend-database wise, there&#8217;s not upgrade errors.<\/p>\n<p>At this time the test only validates the database changes are applicable; in the future this may expand onto more elaborate tests.<\/p>\n<h3>System testing<\/h3>\n<p>I&#8217;m most excited about this one. Taking ideas from our approach to <a href=\"http:\/\/code.openark.org\/blog\/mysql\/using-dbdeployer-in-ci-tests\">testing gh-ost with dbdeployer<\/a>, I created <a href=\"https:\/\/github.com\/openark\/orchestrator-ci-env\">https:\/\/github.com\/openark\/orchestrator-ci-env<\/a>, which offers a full blown testing enviroment for <code>orchestrator<\/code>, including a MySQL replication topology (courtesy <a href=\"https:\/\/www.dbdeployer.com\/\">dbdeployer<\/a>), Consul, HAProxy and more.<\/p>\n<p>This CI testing environment can also serve as a playground in your local docker setup, see shortly.<\/p>\n<p>The <a href=\"https:\/\/github.com\/openark\/orchestrator\/tree\/master\/tests\/system\">system tests suite<\/a> offers full blown cluster-wide operations such as graceful takeovers, master failovers, errant GTID transaction analysis and recovery and more. The suite utilizes the CI testing environment, breaks it, rebuilds it, validates it&#8230; Expects specific output, expects specific failure messages, specific analysis, specific outcomes.<\/p>\n<p>As example, with the system tests suite, we can test the behavior of a master failover in a multi-DC, multi-region (obviously simulated) environment, where a server marked as &#8220;candidate&#8221; is lagging behind all others, with strict rules for cross-site\/cross-region failovers, and still we wish to see that particular replica get promoted as master. We can test not only the topology aspect of the failover, but also the failover hooks, Consul integration and its effects, etc.<\/p>\n<h2>Development<\/h2>\n<p>There&#8217;s now multiple options for developers\/contributors to build or just try out <code>orchestrator<\/code>.<\/p>\n<h3>Build on GitHub<\/h3>\n<p>As mentioned earlier, you actually don&#8217;t need a development environment. You can use <code>orchestrator<\/code> CI to build and generate a Linux\/amd64 <code>orchestrator<\/code> binary, which you can download &amp; deploy as you see fit.<\/p>\n<p>I&#8217;ve signed up for the GitHub Codespaces beta program, and hope to make that available for <code>orchestrator<\/code>, as well.<\/p>\n<h3>Build via Docker<\/h3>\n<p><code>orchestrator<\/code> offers various Docker build\/run environments, accessible via the <code>script\/dock<\/code> script:<\/p>\n<ul>\n<li>`script\/dock alpine` will build and spawn `orchestrator` on a minimal <code>alpine<\/code> linux<\/li>\n<li>`script\/dock test` will build and run the same CI tests (unit, integration) as mentioned earlier, but on your own docker environemtn<\/li>\n<li>`script\/dock pkg` will build and generate `.rpm` and `.deb` packages<\/li>\n<\/ul>\n<h3>CI environment: the &#8220;full orchestrator experience&#8221;<\/h3>\n<p>This is the <code>orchestrator<\/code> amusement park. Run <code>script\/dock system<\/code> to spawn the aforementioned CI environment used in system tests, and on top of that, an <code>orchestrator<\/code> setup fully integrated with that system.<\/p>\n<p>So that&#8217;s an <code>orchestrator<\/code>-MySQL topology-Consul-HAProxy setup, where <code>orchestrator<\/code> already has the credentials for, and pre-loads the MySQL topology, pre-configured to update Consul upon failover, HAProxy config populated by <code>consul-template<\/code>, heartbeat injection, and more. It resembles the <a href=\"https:\/\/github.blog\/2018-06-20-mysql-high-availability-at-github\/\">HA setup at GitHub<\/a>, and in the future I expect to provide alternate setups (on top).<\/p>\n<p>Once in that docker environment, one can try running relocations, failovers, test <code>orchestrator<\/code>&#8216;s behavior, etc.<\/p>\n<h2>Community<\/h2>\n<p>GitHub recently announced <a href=\"https:\/\/github.blog\/2020-05-06-new-from-satellite-2020-github-codespaces-github-discussions-securing-code-in-private-repositories-and-more\/#discussions\">GitHub Discussions<\/a> ; think a stackoverflow like place within one&#8217;s repo to ask questions, discuss, vote on answers. It&#8217;s expected to be available this summer. When it does, I&#8217;ll encourage the community to use it instead of today&#8217;s <a href=\"https:\/\/groups.google.com\/forum\/#!forum\/orchestrator-mysql\">orchestrator-mysql<\/a> Google Group and of course the many questions posted as Issues.<\/p>\n<p>There&#8217;s been a bunch of PRs merged recently, with more to come later on. I&#8217;m grateful for all contributions. Please understand if I&#8217;m still slow to respond.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recent focus on development &amp; testing yielded with new orchestrator environments and offerings for developers and with increased reliability and trust. This post illustrates the new changes, and see Developers section on the official documentation for more details. Testing In the past four years orchestrator was developed at GitHub, and using GitHub&#8217;s environments for testing. [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"enabled":false},"version":2}},"categories":[5],"tags":[136,135,123,57,108,131],"class_list":["post-8077","post","type-post","status-publish","format-standard","hentry","category-mysql","tag-dbdeployer","tag-docker","tag-github","tag-open-source","tag-orchestrator","tag-testing"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p2bZZp-26h","_links":{"self":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/8077","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/comments?post=8077"}],"version-history":[{"count":20,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/8077\/revisions"}],"predecessor-version":[{"id":8096,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/8077\/revisions\/8096"}],"wp:attachment":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/media?parent=8077"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/categories?post=8077"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/tags?post=8077"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}