At the very same day I started writing about it, Baron published What's wrong with MMM?. I wish to present the problems I encountered and the reasons I find MMM is flawed. In a period of two weeks, two different deployments presented me with 4 crashes, in 3 different scenarios.

In all the following scenarios, there is an Active/Passive Master-Master deployment, with one VIP (virtual IP) set for writer role, one VIP set for reader role.

Problem #1: unjustified failover, broken replication

Unjustified failover must be the common scenario. It's also a scenario I can live with. A few seconds of downtime are OK with me once in a couple of months.

But on two different installations, a few days apart, I had two seemingly unjustified failovers followed by a troubling issue: replication got broken.

How broken? The previously active master, now turned inactive, suddenly changed master position to a roughly 10 days old position. I don't keep master logs for 10 days, so this led to an immediate replication fail.

Now, I cannot directly point my finger at MMM, but:

• There was no power failure
• MySQL daemon did not go down
• Replication was just fine both ways up to the failover moment
• There was no human intervention on this (myself and once more person had access at that time).

I know the above since I've got it all monitored. So I can't blame it on “replication not synching master.info file to disk when power went down”. I confess, this is very suspicious; but, twice, on two different deployments...

So much for suspicions. Now for the smoking guns.

Problem #2: hanging master, no failover

The active master went down. Either hardware or software problem caused it to freeze. It was not executing its chores; it became inaccessible by TCP/IP.

But not just inaccessible: freezing inaccessible. If you were to attempt an SSH connection, the connection would just hang; not refused. The SSH client would not terminate in any reasonable time.

Ahem, time to do failover?

Apparently not. Phones start ringing. Emails sent. Time for manual intervention. But, what does the MMM monitor have to say? Nothing. It's frozen. Now, I didn't read the source code; I'm not even competent with PERL; but is seems to me like the monitor daemon works single threaded: it attempts to connect all hosts on the same thread. But, connecting to active master makes for hanging connection, so the entire monitor goes down. Impossible to stop it gracefully, I had to kill it. I had the choice of reconfiguring it to ignore active master, but decided to try starting it up again. I had to do it twice before it started acting sanely again, and realized it was time for failover.

Why is this bad? Assuming my analysis is correct, this is a major design flaw. You must never do a single threaded monitoring on a multiple-machine deployment.

Problem #3: no servers

System is down! No access to the database. What does MMM monitor have to say?

Both machines are HARD_OFFLINE.

Ahem. Both machines are up and running; they are both replicating each other. They are both accessible. MySQL is accessible.

But neither machine has any VIP associated with.

Does it matter whether both agents fail to realize their associated MySQL servers were actually up and running, or whether the monitor fail to receive that information? It does not. MMM should not have removed all VIPs. OK, suppose it believes the two machines are down. So what? Just throw all VIPs on one of the machines. If it's inaccessible, then what's wrong with that? (assuming the previous problem never existed, that is).

You should always have some machine associated with the VIPs.

Temper down

None of the above is meant as offense to the creators of MMM, whom I greatly respect. This isn't an easy problem to solve, and it should be obvious there's no 100% guaranteed solution. But, for myself, I will not be using MMM as it stands right now anymore.

Wednesday

• BoF: Replication, Lars Thalmann, Mat Keep (Oracle)

Lars Thallman presented the MySQL 5.5 replication features, as well as the expected 5.6 features. Among other features, one could notice parallel replication, binlog checksums, sub-second resolution and more. There was an open discussion about these features, asking for comments; looking for new ideas and suggestion from the audience.

I can't possibly cover it all. I'll note two discussion I participated in, and which have interested me. This also serves for noting down to myself my ideas and thoughts.

Parallel replication

As the trend goes, parallel replication will be implemented by having a thread per schema. My own contribution to the discussion is that with such paradigm, the slave may be at any state inconsistent with the master. That is, at any point in time it can be in such state that has never existed in any point in time on the master. This is because it is possible that one or more of the replication threads executes much faster than one or more of the others.

We have discussed the meaning of STOP SLAVE. I have suggested that STOP SLAVE works in such way that it stops in a consistent slave. Lars suggested there could be normal STOP SLAVE and STOP SLAVE CONSISTENT.

It has been noted that we may not care about the slave being inconsistent, as the prerequisite for parallel replication is that the distinct schemata are independent of one another. After some thought, I disagree. Today parallel replication may be implemented on different schemata; tomorrow - based on finer parameters. I think it's best to prepare for the inevitable future. Besides, STOP SLAVE INCONSISTENT is such an easy feature to implement!

Slave analysis

I also had a long time feature request. While writing management tools, I realized it is difficult to learn the topology of a replication environment by simple interrogation. If you connect to a master, you can either issue SHOW PROCESSLIST to detect the slaves, or SHOW SLAVE HOSTS.

The former does not tell you the port on which the slaves are listening. The latter requires that you configure report_host and report_port properly.

What I would like to see is a SHOW SLAVES STATUS command (or similar INFORMATION_SCHEMA table) on the master. Such that it lists connected slaves, one per row, with the following details:

• Slave_host (this is easy, already exists in SHOW PROCESSLIST)
• Slave_port (slave should send this kind of information to master upon first connect)
• Slave_master_log_file (last log file requested by slave)
• Slave_master_log_pos (last log pos requested by slave)
• Slave_seconds_behind_master (slave should report this upon each request)
• Slave_server_id (slave should send this upon connect)

By reading Slave_host, Slave_port & Slave_server_id, it makes it easy for us to build replication topology. It's a single operation to detect a master's slaves. It takes recursive steps to build entire topology.

By reading the Slave_master_log_file, Slave_master_log_pos, we can easily determine that binary logs up to that point have already been written to the slave's relay log. This makes it easier to decide how to purge master logs. Slave_seconds_behind_master completes the replication picture.

I've written all about it before on Dangers of skip-grant-tables.

Darren's 1st advice (look for password ini files, scripts, etc.) is a very good one. One password that can always be looked up in files is the replication's password.

Replication's password is easily forgotten: you only set it once and never use it again; never script it nor manually login with. When setting up new slaves, though, you suddenly need it.

Apparently not many realize that the replication password is written in plaintext in the master.info file. This file tells the slave all about it's master connection: host, port, user & password are all there for you to read.

So your ibdata1 file is some 150GB, and it won't reduce. Really, it won't reduce. You set innodb_file_per_table=1, do ALTER TABLE t ENGINE=InnoDB (optionally ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8), and you get all your tables in file-per-table .ibd files.

But the original ibdata1 file is still there. It has to be there, don't delete it! It contains more than your old data.

InnoDB tablespace files never reduce in size, it's an old-time annoyance. The only way to go round it, if you need the space, is to completely drop them and start afresh. That's one of the things so nice about file-per-table: an ALTER TABLE actually creates a new tablespace file and drops the original one.

The procedure

The procedure is somewhat painful:

• Dump everything logically (either use mysqldump, mk-parallel-dump, or do it your own way)
• Erase your data (literally, delete everything under your datadir)
• Generate a new empty database
• Load your dumped data.

Using replication

Replication makes this less painful. Set up a slave, have it follow up on the master.

• Stop your slave.
• Make sure to backup the replication position (e.g. write SHOW SLAVE STATUS on a safe location, or copy master.info file).
• Work out the dump-erase-generate-load steps on the slave.
• Reattach the slave to the master using saved data.

For this to succeed you must keep enough binary logs on the master for the entire dump-load period, which could be lengthy.

Upgrading to barracuda

If you wish to upgrade your InnoDB tables to Barracuda format, my advice is this:

1. Follow the steps above to generate a file-per-table working slave
2. Stop the slave
3. Configure skip_slave_start
4. Restart MySQL
5. One by one do the ALTER TABLE into Barracuda format (ROW_FORMAT=COMPACT or ROW_FORMAT=COMPRESSED)

Note that if you're about to do table compression, the ALTER statements become considerably slower the better the compression is.

If your dataset is very large, and you can't keep so many binary logs, you may wish to break step 5 above into:

• ALTER a large table
• Restart MySQL
• Start slave, wait for it to catch up
• Restart MySQL again

and do the same for all large tables.

Why all these restarts?

I've been upgrading to Barracuda for a long time now. I have clearly noticed that ALTER into a COMPRESSED format works considerably slower after the slave has done some "real work". This in particular relates to the last "renaming table" stage. There was a bug with earlier InnoDB plugin versions which made this stage hang. It was solved. But it still takes some time for this last, weird stage, where the new replacement table is complete, and it's actually been renamed in place of the old table, and the old table renamed into something like "#sql-12345.ibd", and all that needs to be done is have it dropped, and... Well, it takes time.

My observation is it works faster on a freshly started server. Which is why I take the bother to restart MySQL before each large table conversion.

The openark kit is a set of utilities for MySQL. They solve everyday maintenance tasks, which may be complicated or time consuming to work by hand.

It's been a while since the last announced release. Most of my attention was on mycheckpoint, building new features, writing documentation etc. However my own use of openark kit has only increased in the past few months, and there's new useful solutions to common problems that have been developed.

I've used and improved many tools over this time, but doing the final cut, along with proper documentation, took some time. Anyway, here are the highlights:

New tool: oak-hook-general-log

oak-hook-general-log hooks up a MySQL server and dumps the general log based on filtering rules, applying to query role or execution plan. It is possible to only dump connect/disconnect entries, queries which make a full table scan, or use temporary tables, or scan more than X number of rows, or...

I'll write more on this tool shortly.

New tool: oak-prepare-shutdown

This tool makes for an orderly and faster shutdown by safely stopping replication, and flushing InnoDB pages to disk prior to shutting down (keeping server available for connections even while attempting to flush dirty pages to disk). A typical use case would be:

oak-prepare-shutdown --user=root --ask-pass --socket=/tmp/mysql.sock && /etc/init.d/mysql stop

New tool: oak-repeat query

oak-repeat-query repeats executing a given query until some condition holds. The condition can be:

• Number of given iterations has been reached
• Given time has elapsed
• No rows have been affected by query

The tool comes in handy for cleanup jobs, warming up caches, etc.

New tool: oak-get-slave-lag

This simple tool just returns the number of seconds a slave is behind master. But it also returns with an appropriate exit code, based on a given threshold: 0 when lag is good, 1 (error exit code) when lag is too great or slave fails to replicate.

This tool has been used by 3rd party applications, such as a load balancer, to determine whether a slave should be accessed.

Updated tool: oak-chunk-update

This extremely useful utility breaks down very long queries into smaller chunks. These could be queries which should affect a huge amount of rows, or queries which cannot utilize an index.

Updates to the tool include limiting the range of rows the tool scans, by specifying start and stop position (either by providing constant values or by SELECT query). Also added is auto-termination when no rows are found to be affected. Last, it is possible to override INFORMATION_SCHEMA lookup by explicitly specifying chunking key.

This tool works great for your daily/weekly/monthly batch jobs; in creating DWH tables; populating new columns; purging old entries; clearing data based on non-indexed values; generating summary tables; and more.

Frozen tool: oak-apply-ri

I haven't been using this tool for a while. The main work down by this tool can be done with oak-chunk-update. There are some additional safety checks oak-apply-ri provides; I'm thinking over if they justify the tool's existence.

Frozen tool: oak-online-alter-table

With the appearance of Facebook’s Online Schema Change (OSC) tool, which derives from oak-online-alter-table, I'm not sure I will continue developing the tool. I intend to wait for general feedback on OSC before making a decision.

Documentation

Documentation is now part of openark kit's SVN repository.

Download

The openark kit project is currently hosted by Google Code. Downloads are available at the Google Code openark kit project page.

Downloads are available in the following packaging formats:

• .deb package, to be installed on debian, ubuntu and otherwise debian based distributions.
• .rpm package, architecture free (noarch), for RPM supporting Linux distributions such as RedHat, Fedora, CentOS etc.
• .tar.gz using python's distutils installer.
• source, directly retrieved from SVN or from above python package.
• Some distribution specific RPM packages, courtesy Lenz Grimmer.

Feedback

Your feedback is welcome! I may not always respond promptly; and I confess that some bugs were left open for more than I would have liked them to. I hope to make for good quality of code, and bug reporting is one major factor you can control.

The two IPs for the reader role

A simplified excerpt from the mmm_common.conf sample configuration file, as can be found on the project's site and which is most quoted:

...
<host db1>
  ip                      192.168.0.11
  mode                    master
  peer                    db2
</host>

<host db2>
  ip                      192.168.0.12
  mode                    master
  peer                    db1
</host>
...
<role writer>
  hosts                   db1, db2
  ips                     192.168.0.100
  mode                    exclusive
</role>

<role reader>
  hosts                   db1, db2
  ips                     192.168.0.101, 192.168.0.102
  mode                    balanced
</role>

In the above setup db1 & db2 participate in master-master active-passive replication. Whenever you need to write something, you use 192.18.0.100, which is the virtual IP for the writer role. Whenever you need to read something, you use either 192.168.0.101 or 192.168.0.102, which are the virtual IPs of the two machines, this time in read role. Logic says one wishes to distribute reads between the two machines.

One IP for reader role

I have a few cases where the above setup is not satisfactory: there is a requirement to know the IP of the passive (read-only) master. Reason? There are queries which we only want to execute on the slave (reporting, long analysis), and only execute on the active master when this isn't possible. Sometimes we might even prefer waiting for a slave to come back up rather than execute a query on the master.

This may involve an application level solution, or a connection-pool level solution ("get me a slave's connection, or, if that's not possible, get me the master's").

Anyway, neither 192.168.0.101 nor 192.168.0.102 relate to a particular machine's role status. That is, the fact that one of the machines is in writer mode or not does not affect these virtual IPs.

The solution is a minor change to the configuration file. Real minor:

<role reader>
  hosts                   db1, db2
  ips                     192.168.0.101
  mode                    balanced
</role>

In this new setup the two nodes compete for a single reader role virtual IP. There is no 192.168.0.102 anymore. Although it does not reflect from the configuration file, it turns out MMM acts in a smart way; the way you would expect it to run.

There is nothing to suggest in the above that the IPs 192.168.0.100 & 192.168.0.101 will be distributed between the two machines. But you would like them to. And MMM does that. It makes sure that, if possible, one of the machines (say db1) gets the writer role, hence 192.168.0.100, and the other (db2) the reader role, hence 192.168.0.101.

Moreover, it prefers that situation over a current known situation: say db1 went down. The writer role moves to db2. When db1 is up again, MMM acts smartly: it does not give it back the writer role (since moving the active master around is costly, after all), but does give it the reader role, along with the 192.168.2.101 IP. So it takes care not to leave a server without a role, while preferring to move the writer role as little as possible.

As I'll show, DROP operations are significantly simpler than creation operations. Interestingly, it's the same as in life.

DROP COLUMN

A column turns to be redundant, unused. Before it is dropped from the database, we must ensure no one is using it anymore. The steps are:

1. App: V1 -> V2. Remove all references to column; make sure no queries use said column.
2. DB: V1 -> V2 (possibly failover from M1 to M2), change is DROP COLUMN.

DROP INDEX

A possibly simpler case here. Why would you drop an index? Is it because you found out you never use it anymore? Then all you have to do is just drop it.

Or perhaps you don't need the functionality the index supports anymore? Then first drop the functionality:

1. (optional) App: V1 -> V2. Discard using functionality which relies on index.
2. DB: V1 -> V2 (possibly failover from M1 to M2), change is DROP INDEX. Check out InnoDB Plugin here.

DROP UNIQUE INDEX

When using Master-Slave failover for table refactoring, we're now removing a constraint from the slave. Since the master is more constrained than the slave, there is no problem here. It's mostly the same as with a normal DROP INDEX, with a minor addition:

1. (optional) App: V1 -> V2. Discard using functionality which relies on index.
2. DB: V1 -> V2 (possibly failover from M1 to M2), change is DROP INDEX.
3. (optional) App: V2 -> V3. Enable functionality that inserts duplicates.

DROP FOREIGN KEY

Again, we are removing a constraint.

1. DB: V1 -> V2 (possibly failover from M1 to M2), change is DROP INDEX.
2. (optional) App: V2 -> V3. Enable functionality that conflicts with removed constraint. I mean, if you really know what you are doing.

DROP TABLE

The very simple steps are:

1. App: V1 -> V2. Make sure no reference to table is made.
2. DB: V1 -> V2. Issue a DROP TABLE.

With ext3 dropping a large table is no less than a nightmare. Not only does the action take long time, it also locks down the table cache, which very quickly leads to having dozens of queries hang. xfs is a good alternative.

Conclusion

We looked at single table operations, coupled with application upgrades. By carefully looking at the process breakdown, multiple changes can be addressed with ease and safety. Not all operations are completely safe when used with replication failover. But they are mostly safe if you have some trust in your code.

I will break down the discussion into types of database refactoring operations, and I will limit to single table refactoring. The discussion will try to understand the need for refactoring and will dictate the steps towards a successful upgrade.

Reader prerequisites

I will assume MySQL to be the underlying database. To take a major component out of the equation: we may need to deal with very large tables, for which an ALTER command may take long hours. I will assume familiarity with Master-Master (Active-Passive) replication, with possible use of MMM for MySQL. When I describe "Failover from M1 to M2", I mean "Make the ALTER changes on M2 (passive), then switch your application from M1 to M2 (change of IPs, VIP, etc.), promoting M2 to active position, then apply same changes on M1 (now passive) or completely rebuild it".

Phew, a one sentence description of M-M usage...

I also assume the reader's understanding that a table's schema can be different on master & slave, which is the basis for the "use replication for refactoring" trick. But it cannot be too different, or, to be precise, the two schemata must both support the ongoing queries for the table.

A full discussion of the above is beyond the scope of this post.

Types of refactoring needs

As I limit this discussion to single table refactoring,we can look at major refactoring operations and their impact on application & upgrades. We will discuss ADD/DROP COLUMN, ADD/DROP INDEX, ADD/DROP UNIQUE INDEX, ADD/DROP FOREIGN KEY, ADD/DROP TABLE.

We will assume the database and application are both in Version #1 (V1), and need to be upgraded to V2 or greater.

ADD INDEX

Starting with the easier actions. Why would you add an index? Either:

1. There is some existing query which can be optimized by the new query
2. Or there is some new functionality which issues a query for which the new index is required.

Adding an index is an easy action in that the table's data does not really change.

In case #1, all you need to do is to add the new index (if the table is large, fail over from M1 to M2). There is no application upgrade, so all that happens is that the database upgrades V1 -> V2.

In case #2, the database must be prepared with new schema before the new functionality/query is introduced (since it depends on the existence of the index). The steps, therefore, are:

1. DB: V1 -> V2 (possibly failover from M1 to M2)
2. (Sometime later) App: V1 -> V2. Application will issue queries which utilize the new index.

The application does not have to be upgraded at the same instant the DB gets upgraded. In fact, we'll see that this is a typical scenario: we can separate upgrades into smaller steps, which allow for time lapse. One could work out steps 1 & 2 together, but that would take an extra effort.

ADD COLUMN

This must be one of the most common table schema upgrades: a new property is needed on the application side. It must be supported by the database. Perhaps a new field in some Java Object, with Hibernate mapping that field onto a new column. Or maybe the new column is there for purpose of de-normalization.

This is also a more complicated task. Let's look at the required steps:

1. DB: V1 -> V2 (possibly failover from M1 to M2), change is ADD COLUMN.
2. App: V1 -> V2. Change is: provide column value for newly INSERTed rows.
3. If needed, retroactively update column values for all pre-existing rows.
4. App: V2 -> V3. Application begins to use (read, SELECT) new column.

The above procedure assumes that the new column must have some calculated value. A 10-million rows table must now be updated, to have the correct values filled in. So we ask of the application to start filling in data for new rows, which makes the invalid row set static. We can just take a "from row" and a "to row" and fill in the missing column's value for those rows. Only when all rows contain valid values can we let the application start using that row. This makes for two application upgrades.

If you're content with just a static DEFAULT value, then step 3 can be skipped, and step 4 can be merged with step 2.

ADD UNIQUE INDEX

This is an altogether different case than the normal ADD INDEX, even though they may seem similar. And the case is particularly different when using Master-Slave failover for rebuilding the table.

Consider the case where we add a UNIQUE INDEX on a slave. Some INSERT query executes on the master, successfully, and is logged to the binary log. The slave picks it up, tries to execute it, to find that it fails on a DUPLICATE KEY error.

The UNIQUE INDEX is a constraint, and it makes the slave more constrained than the master. This is a delicate situation. Here how to (mostly) work it out:

1. App: V1 -> V2. Change INSERT queries on relevant table to INSERT IGNORE or REPLACE queries, whichever is more appropriate.
2. DB: V1 -> V2 (possibly failover from M1 to M2), change is ADD UNIQUE KEY (and while at it, a tip: are you aware of ALTER IGNORE TABLE?)

The change of query ensures that the query will succeed on the slave (either by silently doing nothing or by actually replacing content). It also means that the slave can now have different data than the master. Of course, it you trust your application to never INSERT duplicates, you can sleep better.

We do not handle UPDATE statements here.

ADD CONSTRAINT FOREIGN KEY

As with ADD UNIQUE INDEX, there is a new constraint here. A slave becomes more constrained than the master. But we now have to make sure INSERT, UPDATE and DELETE statements all go peacefully (well, it also depends on the type of ON DELETE and ON UPDATE property of the FK).

The steps would be:

1. DB: V1 -> V2 (possibly failover from M1 to M2), change is ADD CONSTRAINT FOREIGN KEY.

And then cross your fingers or have trust in your application. If the table is small enough, one does not have to use replication to do the refactoring, and life is simpler. Just execute the ALTER on the active master, and continue with your life.

CREATE TABLE

This is a simple case, since the table is new. The steps are:

1. DB: V1 -> V2 (no need to use slaves here)
2. App: V1 -> V2. Application will start using new table.

Conslusion

Having such steps formalized help with development management and database management. It makes clear what is expected of the application, and what is expected of the database. The breaking down of these operations into sequential steps allows us to work more slowly; make preparation work; work within our own working hours; get a chance to see the family.

In this post we took a look at "creation" refactoring changes. New columns, new keys, new constraints. In the next part of this article, we'll discuss DROP operations.

It does not explain how to create replicating slaves. See How To Setup Replication for that. However, not all configuration options are well understood, and their roles in varying architectures can change.

Here are the settings for a basic Master/Slave(s) replication architecturee.

Essential

• log-bin: enable binary logs on the master. Replication is based on the master logging all modifying queries (INSERT/CREATE/ALTER/GRANT etc.), and the slaves being able to replicate them.
• server-id: each machine must have a unique server-id. A slave will not replay queries originating from a server with the same server-id as its own.
• GRANT: grant a user with REPLICATION SLAVE. The host list must include all replication slave hosts.
• expire-logs-days: automatically clean up master's binary logs older than given value. By default, binary logs are never removed.

When working with Master/Slaves replication, one should be prepared to master failure and slave promotion to master. It may be desirable to identify a particular slave as primary candidate for promotion.

Just setting up the log-bin will yield with warnings in the MySQL's error log. The binary logs are named, by default, after the host's name. If that should change - MySQL will not be able to find the binary logs anymore (expecting a name which does previous logs did not use). It is therefore recommended to use:

log-bin=mychachine-bin

or

log-bin=mysql-bin

Essential/Optional

• log-bin: enable on a slave, so that in case it is promoted to master, the rest of the slaves can replicate using its binary logs. Enabling binary logging cannot be done on a live server: this parameter requires MySQL restart.
• GRANT: include the master's host, so that when a slave promotes to master, the master can become a slave and continue replicating.
• log-slave-updates: together with log-bin, enable on slave so that master's binary logs are propagated and logged by the slave. This is required if the slave takes the role of a master in a chained replication setup.
• expire-logs-days: set this flag on slave as well [tnx Sheeri].
• read-only: set on slave(s). Refuses any modifying query (INSERT, DELETE, ALTER, DROP etc.) for non-SUPER privileged users [tnx Ryan].
• sync-binlog: flush binary log to disk per transaction commit. Use this on master for safer replication; however note that increased I/O is expected [tnx Harrison].

Extra

• report-host, report-port: the host and port identifying the slave when looking at SHOW SLAVE HOSTS on master. Set this up on all hosts. See further discussion here.
• max-binlog-size: the maximum size for a binary log / relay log file, after which it is rotated.

Expert

• binlog-do-db, binlog-do-table, replicate-do-db, ...: filter queries by either not writing them to binary log, or not reading them from the logs.

The reason I list the above as "Expert" is not because one must have a super-brain to set them up. That part is easy enough. But they lead to some dangerous situations, sometimes seemingly harmless. It takes great care to control the application and developers from creating those situations. See documentation here. See also discussion here and here.

Using binlog-do-db is dangerous to your replication. It means the master will not write to binary logs any statement not in the given database.

Ahem. Not exactly. It will not write to binary logs any statement which did not originate from the given database.

Which is why a customer, who was using Toad for MySQL as client interface to MySQL, and by default connected to the mysql schema, did not see his queries being replicated. In fact, he later on got replication errors. If you do:

USE test;
INSERT INTO world.City VALUES (...)

Then the statement is assumed to be in the test database, not in the world database.

Slightly better is using replicate-do-db on the slave machines. At least we allow the master to write everything. But still, for the same reasons, slaves may fail to repeat a perfectly valid query, just because it has been issued in the context of the wrong database. replicate-ignore-db is somewhat safer yet, but the trap is still there.

My advice is that replication should replicate everything. Make sure you and everyone else you work with understand the implications of binlog-do-db and replicate-do-db before implementing it.