Skip to content

# Tag: Open Source

## Orchestrator 1.4.340: GTID, binlog servers, Smart Mode, failovers and lots of goodies

MySQL Open Source orchestrator Pseudo GTID Replication September 1, 2015September 11, 2015

Orchestrator 1.4.340 is released. Not quite competing with the MySQL latest changelog, and as I haven’t blogged about orchestrator featureset in a while, this is a quick listing of orchestrator features available since my last publication:

- Supports GTID (Oracle & MariaDB)
  - GTID still not being used in automated recovery — in progress.
  - enable-gtid, disable-gtid, skip-query for GTID commands
- Supports binlog servers (MaxScale)
  - Discovery & operations on binlog servers
  - Understanding slave repositioning in a binlog-server architecture
- Smart mode: relocate & relocate-below commands (or Web/GUI drag-n-drop) let orchestrator figure out the best way of slave repositioning. Orchestrator picks from GTID, Pseudo GTID, binlog servers, binlog file:pos math (and more) options, or combinations of the above. Fine grained commands still there, but mostly you won’t need them.
- Crash recoveries (did you know orchestrator does that?):
  - For intermediate master recovery: improved logic in picking the best recovery plan (prefer in-DC, prefer promoting local slave, supporting binlog server topologies, …)
  - For master recovery: even better slave promotion; supports candidate slaves (prefer promoting such slaves); supports binlog server shared topologies
  - Better auditing and logging of recovery cases
  - Better analysis of crash scenarios, also in the event of lost VIPs, hanging connections; emergent checks in crash suspected scenarios
  - recover-lite: do all topology-only recovery steps, without invoking external processes
- Better browser support: used to only work on Firefox and Chrome (and the latter has had issues), the Web UI should now work well on all browsers, at the cost of reduced d3 animation. More work still in progress.
- Faster, more parallel, less blocking operations on all counts; removed a lots of serialized code; less locks.
- Web enhancements
  - More verbose drag-n-drop (operation hint; color hints)
  - Drag-n-drop for slaves-of-a-server
  - Replication/crash analysis dashboard
- Pools: orchestrator can be told about instance-to-pool association (submit-pool-instances command)
  - And can then present pool status (web)
  - Or pool hints within topologies (web)
  - Or queried for all pools (cluster-pool-instances command)
- Other:
  - Supports MySQL 5.7 (tested with 5.7.8)
  - Configurable graphite path for metrics
  - –noop flag; does all the work except for actually changing master on slaves. Shows intentions.
  - Web (or cli which-cluster-osc-slaves command) provide list of control slaves to use in pt-osc operation
  - hostname-unresolve: force orchestrator to unresolve a fqdn into VIP/CNAME/… when issuing a CHANGE MASTER TO
- 3rd party contributions (hey, thanks!) include:
  - More & better SSL support
  - Vagrant templates
- For developers:
  - Orchestrator now go-gettable. Just go get github.com/outbrain/orchestrator
  - Improved build script; supports more architectures

Continue reading » “Orchestrator 1.4.340: GTID, binlog servers, Smart Mode, failovers and lots of goodies”

## “awesome-mysql” curated list created, open for pull requests

MySQL community Open Source tools April 28, 2015

Following up on popular “awesome-*” lists (e.g. awesome-python, awesome-golang etc.), I’ve created the awesome-mysql curated list.

This is a list of technologies (and resources) in and around MySQL, and means to serve as a place to find reliable software and info. I recently happened to notice there are some tools I’m familiar with that are unknown to others; tools unknown to me that are in good use.

The list is naturally and intentionally incomplete. I wish this to be a community based creation; so I put some categories and some tools. I left many out, deliberatey. Please assist by filling in the missing projects, tools, libraries! Additions gladly accepted via pull-requests. Do note the contribution guidelines (somewhat lengthy, I apologize).

I will moderate FUDs, promotional, commercials etc., and otherwise it may take some days for me to merge requests.

The work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.

## Percona Live 2015: Reflections; the Apache CCLA offer

MySQL community Open Source Opinions secondary April 18, 2015April 18, 2015

Facebook, Google, Twitter, LinkedIn, Alibaba, MariaDB, Percona team up and offer Oracle all public changes under the Apache CCLA

Read again please.

My one word summary of this is: Romantic. In the most positive sense.

Disclaimer: I am not a lawyer; this is my understanding of the current status and of the offer.

Summarizing the deal: the teams participating with WebScaleSQL would like to push code upstream. Current legal issues limit their options. Existing patches/contributions from Percona & MariaDB are licensed by GPLv2, which Oracle cannot import as it distributes a commercial, closed source, edition, in addition to its open source MySQL community edition.

So what happens is that there is a lot of free code, great patches, new features out there, that are only available via MariaDB or WebscaleSQL or Percona Server, but not in the Oracle MySQL code base. This, in turn, means Oracle re-implements many features originating from said companies. And, more importantly, said companies need to routinely rebase their code on new Oracle releases, repeating tedious work.

The offer is that Oracle agrees to the Apache CCLA as a license by which it would be able to incorporate contributions. Oracle would then be able to use incorporated code in both open source and commercial edition. Oracle will choose what code to incorporate; hopefully many patches will be accepted upstream, and the community will benefit from a rich featureset, rapid developed MySQL server.

Clearly a lot of work, persuasion, lawyer time, discussions etc. have been invested in this effort. I would like to add my humble +1/like/favorite/whathaveyou. You may add yours by letting Oracle know your opinion on the subject. Media tools are great for this.

## Speaking at Percona Live: Pseudo GTID and Easy Replication Topology Management

MySQL Open Source orchestrator PerconaLive Pseudo GTID Replication secondary March 31, 2015April 30, 2015

In two weeks time I will be presenting Pseudo GTID and Easy Replication Topology Management at Percona Live. From the time I submitted the proposal a LOT has been developed, experimented, deployed and used with both Pseudo GTID and with orchestrator. In my talk I will:

- Suggest that you skip the “to GTID or not to GTID” question and go for the lightweight Pseudo GTID
- Show how Pseudo GTID is used in production to recover from various replication failures and server crashes
- Do an outrageous demonstration
- Tell you about 50,000 successful experiments and tests done in production
- Show off orchestrator and its support for Pseudo GTID, including automated crash analysis and recovery mechanism.

I will further show how the orchestrator tooling makes for a less restrictive, more performant, less locking, non-intrusive, trusted and lightweight replication topology management solution. Continue reading » “Speaking at Percona Live: Pseudo GTID and Easy Replication Topology Management”

## Orchestrator 1.2.9 GA released

MySQL Open Source orchestrator Replication December 18, 2014

Orchestrator 1.2.9 GA has been released. Noteworthy:

- Added “ReadOnly” (true/false) configuration param. You can have orchestrator completely read-only
- Added “AuthenticationMethod”: “multi”: works like BasicAuth (your normal HTTP user+password) only it also accepts the special user called “readonly”, which, surprise, can only view and not modify
- Centralized/serialized most backend database writes (with hundreds/thousands monitored servers it was possible or probable that high concurrency led to too-many-connections openned on the backend database).
- Fixed evil evil bug that would skip some checks if binary logs were not enabled
- Better hostname resolve (now also asking MySQL server to resolve hostname; resolving is cached)
- Pseudo-GTID (read here, here, here) support now considered stable (apart from being tested it has already been put to practice multiple times in production at Outbrain, in different planned and unplanned crash scenarios)

I continue developing orchestrator as free and open source at my new employer, Booking.com.

## Orchestrator 1.2.1 BETA: Pseudo GTID support, reconnect slaves even after master failure

MySQL Open Source orchestrator Pseudo GTID Replication October 27, 2014

orchestrator 1.2.1 BETA is released. This version supports Pseudo GTID, and provides one with powerful refactoring of one’s replication topologies, even across failed instances.

> Depicted: moving a slave up the topology even though its local master is inaccessible

#### Enabling Pseudo-GTID

You will need to:

1. Inject a periodic unique entry onto your binary logs
2. Configure orchestrator to recognize said entry.

#### Pseudo GTID injection example

We will use the event scheduler (must be enabled) to inject an entry every 10 seconds, recognized both in statement-based and row-based replication.

>     create database if not exists meta;

>     drop event if exists meta.create_pseudo_gtid_view_event;

>     delimiter ;;
>     create event if not exists
>       meta.create_pseudo_gtid_view_event
>       on schedule every 10 second starts current_timestamp
>       on completion preserve
>       enable
>       do
>         begin
>           set @pseudo_gtid := uuid();
>           set @_create_statement := concat('create or replace view meta.pseudo_gtid_view as select \'', @pseudo_gtid, '\' as pseudo_gtid_unique_val from dual');
>           PREPARE st FROM @_create_statement;
>           EXECUTE st;
>           DEALLOCATE PREPARE st;
>         end
>     ;;

>     delimiter ;

>     set global event_scheduler := 1;

Make sure to enable event_scheduler in your my.cnf config file.

An entry in the binary logs would look like this: Continue reading » “Orchestrator 1.2.1 BETA: Pseudo GTID support, reconnect slaves even after master failure”

## orchestrator 1.1.18: new features, support for orchestrator-agent

MySQL Open Source orchestrator Replication October 13, 2014

Outbrain‘s orchestrator 1.1.18 is released:

- Support for orchestrator-agent (see announcement): agent pages, support for agent actions, initiation of seeds (provisioning of new/corrupted servers), auditing of seeds.
- Clusters dashboard
- Support for long query auditing
- SSL
- Proxy authentication (e.g. apache2 serving as reverse-proxy with LDAP)
- User control
- Better slave moving rules.

Quick links:

- Get orchestrator 1.1.18
- Read the orchestrator manual
- Check out orchestrator-agent

orchestrator now allows for seeding/provisioning of servers via orchestrator-agent. It communicates with agents executing on the MySQL hosts and coordinate transfer of data between them. orchestrator now supports invocation and auditing of seeding operations, and protects you from breaking your seeds. The orchestrator-agent is a solution to Outbrain’s specific use case, and may not appeal to the greater crowd. Nonetheless it is extendible and is released as open source. Continue reading » “orchestrator 1.1.18: new features, support for orchestrator-agent”

## Announcing orchestrator-agent

MySQL Backup Open Source orchestrator Replication October 13, 2014

orchestrator-agent is a side-kick, complementary project of orchestrator, implementing a daemon service on one’s MySQL hosts which communicates with and accepts commands from orchestrator, built with the original purpose of providing an automated solution for provisioning new or corrupted slaves.

It was built by Outbrain, with Outbrain’s specific use case in mind. While we release it as open source, only a small part of its functionality will appeal to the public (this is why it’s not strictly part of the orchestrator project, which is a general purpose, wide-audience solution). Nevertheless, it is a simple implementation of a daemon, such that can be easily extended by the community. The project is open for pull-requests!

A quick breakdown of orchestrator-agent is as follows:

- Executes as a daemon on linux hosts
- Interacts and invokes OS commands (via bash)
- Does not directly interact with a MySQL server running on that host (does not connect via mysql credentials)
- Expects a single MySQL service on host
- Can control the MySQL service (e.g. stop, start)
- Is familiar with LVM layer on host
- Can take LVM snapshots, mount snapshots, remove snapshots
- Is familiar with the MySQL data directory, disk usage, file system
- Can send snapshot data from a mounted snapshot on a running MySQL host
- Can prepare data directory and receive snapshot data from another host
- Recognizes local/remote datacenters
- Controlled by orchestrator, two orchestrator-agents implement an automated and audited solution for seeding a new/corrupted MySQL host based on a running server.

Continue reading » “Announcing orchestrator-agent”

## zookeepercli: lightweight, powerful, controlled command line client for ZooKeeper

Development Linux command line go Open Source ZooKeeper September 17, 2014

I’m happy to announce the availability of zookeepercli: a lightweight, simple, fast and controlled command line client for ZooKeeper.

zookeepercli allows for:

- Basic CRUD-like operations: `create`, `set`, `delete`, `exists`, `get`, `ls` (aka `children`).
- Extended operations: `lsr` (ls recursive), `creater` (create recursively)
- Well formatted and controlled output: supporting either `txt` or `json` format
- Single, no-dependencies binary file, based on a native Go ZooKeeper library by github.com/samuel/go-zookeeper (LICENSE)

I was dissatisfied with existing command line access to ZooKeeper. Uncontrolled and noisy output as well as large footprint were among the reasons. zookeepercli overcomes the above and provides with often required powers.

Usage samples:

>     $ zookeepercli --servers srv-1,srv-2,srv-3 -c create /demo_only "path placeholder"
>     $ zookeepercli --servers srv-1,srv-2,srv-3 -c create /demo_only/key1 "value1"
>     $ zookeepercli --servers srv-1,srv-2,srv-3 -c create /demo_only/key2 "value2"
>     $ zookeepercli --servers srv-1,srv-2,srv-3 -c create /demo_only/key3 "value3"

>     $ zookeepercli --servers srv-1,srv-2,srv-3 -c ls /demo_only
>     key3
>     key2
>     key1

>     # Same as above, JSON format output:
>     $ zookeepercli --servers srv-1,srv-2,srv-3 --format=json -c ls /demo_only
>     ["key3","key2","key1"]

>     $ zookeepercli --servers srv-1,srv-2,srv-3 -c delete /demo_only/key1
>     $ zookeepercli --servers srv-1,srv-2,srv-3 -c delete /demo_only/key2
>     $ zookeepercli --servers srv-1,srv-2,srv-3 -c delete /demo_only/key3
>     $ zookeepercli --servers srv-1,srv-2,srv-3 -c delete /demo_only

>     # Create a path recursively (auto-generate parent directories if not exist):
>     $ zookeepercli --servers=srv-1,srv-2,srv-3 -c creater "/demo_only/child/key1" "val1"
>     $ zookeepercli --servers=srv-1,srv-2,srv-3 -c creater "/demo_only/child/key2" "val2"

>     $ zookeepercli --servers=srv-1,srv-2,srv-3 -c get "/demo_only/child/key1"
>     val1

>     # This path was auto generated due to recursive create:
>     $ zookeepercli --servers=srv-1,srv-2,srv-3 -c get "/demo_only"
>     zookeepercli auto-generated

>     # ls recursively a path and all sub children:
>     $ zookeepercli --servers=srv-1,srv-2,srv-3 -c lsr "/demo_only"
>     child
>     child/key1
>     child/key2

zookeepercli is released as open source by Outbrain under the Apache 2.0 license.

Quick links:

- Project page
- Pre-built binaries for download
- License

## Orchestrator 1.0.5: refactoring masters, multi-master replication

MySQL Open Source orchestrator Replication August 6, 2014

Outbrain’s orchestrator Version 1.0.5 is released.

> Quick links: Orchestrator Manual, FAQ, Downloads

Orchestrator now supports refactoring of masters via master-master topologies. It now allows promoting slaves as co-masters and detachment of instances from a co-master topology, effectively allowing for replacing an active master.

#### Like this

Drag a master:

Onto one of its slaves: Continue reading » “Orchestrator 1.0.5: refactoring masters, multi-master replication”