The two IPs for the reader role

A simplified excerpt from the mmm_common.conf sample configuration file, as can be found on the project’s site and which is most quoted:

...
<host db1>
  ip                      192.168.0.11
  mode                    master
  peer                    db2
</host>

<host db2>
  ip                      192.168.0.12
  mode                    master
  peer                    db1
</host>
...
<role writer>
  hosts                   db1, db2
  ips                     192.168.0.100
  mode                    exclusive
</role>

<role reader>
  hosts                   db1, db2
  ips                     192.168.0.101, 192.168.0.102
  mode                    balanced
</role>

In the above setup db1 & db2 participate in master-master active-passive replication. Whenever you need to write something, you use 192.18.0.100, which is the virtual IP for the writer role. Whenever you need to read something, you use either 192.168.0.101 or 192.168.0.102, which are the virtual IPs of the two machines, this time in read role. Logic says one wishes to distribute reads between the two machines.

One IP for reader role

I have a few cases where the above setup is not satisfactory: there is a requirement to know the IP of the passive (read-only) master. Reason? There are queries which we only want to execute on the slave (reporting, long analysis), and only execute on the active master when this isn’t possible. Sometimes we might even prefer waiting for a slave to come back up rather than execute a query on the master.

This may involve an application level solution, or a connection-pool level solution (“get me a slave’s connection, or, if that’s not possible, get me the master’s”).

Anyway, neither 192.168.0.101 nor 192.168.0.102 relate to a particular machine’s role status. That is, the fact that one of the machines is in writer mode or not does not affect these virtual IPs.

The solution is a minor change to the configuration file. Real minor:

<role reader>
  hosts                   db1, db2
  ips                     192.168.0.101
  mode                    balanced
</role>

In this new setup the two nodes compete for a single reader role virtual IP. There is no 192.168.0.102 anymore. Although it does not reflect from the configuration file, it turns out MMM acts in a smart way; the way you would expect it to run.

There is nothing to suggest in the above that the IPs 192.168.0.100 & 192.168.0.101 will be distributed between the two machines. But you would like them to. And MMM does that. It makes sure that, if possible, one of the machines (say db1) gets the writer role, hence 192.168.0.100, and the other (db2) the reader role, hence 192.168.0.101.

Moreover, it prefers that situation over a current known situation: say db1 went down. The writer role moves to db2. When db1 is up again, MMM acts smartly: it does not give it back the writer role (since moving the active master around is costly, after all), but does give it the reader role, along with the 192.168.2.101 IP. So it takes care not to leave a server without a role, while preferring to move the writer role as little as possible.

Normally, I would simply:

SELECT @@group_concat_max_len

However, I am using views, where session variables are not allowed. Using a stored function can do the trick, but I wanted to avoid stored routines. So here’s a very simple test case: is the current group_concat_max_len long enough or not? I’ll present the long version and the short version.

The long version

SELECT
  CHAR_LENGTH(
    GROUP_CONCAT(
      COLLATION_NAME SEPARATOR ''
    )
  )
FROM
  INFORMATION_SCHEMA.COLLATIONS;

If the result is 1024, we are in a bad shape. I happen to know that the total length of collation names is above 1800, and so it is trimmed down. Another variance of the above query would be:

SELECT
  CHAR_LENGTH(
    GROUP_CONCAT(
      COLLATION_NAME SEPARATOR ''
    )
  ) = SUM(CHAR_LENGTH(COLLATION_NAME))
    AS group_concat_max_len_is_long_enough
FROM
  INFORMATION_SCHEMA.COLLATIONS;

+-------------------------------------+
| group_concat_max_len_is_long_enough |
+-------------------------------------+
|                                   0 |
+-------------------------------------+

The COLLATIONS, CHARACTER_SETS or COLLATION_CHARACTER_SET_APPLICABILITY tables provide with known to exist variables (assuming you did not compile MySQL with particular charsets). It’s possible to CONCAT, UNION or JOIN columns and tables to detect longer than 1800 characters in group_concat_max_len. I admit this is becoming ugly, so let’s move on.

The short version

Don’t want to rely on existing tables? Not sure what values to expect? Look at this:

SELECT CHAR_LENGTH(GROUP_CONCAT(REPEAT('0', 1025))) FROM DUAL

GROUP_CONCAT doesn’t really care about the number of rows. In the above example, I’m using a single row (retrieved from the DUAL virtual table), making sure it is long enough. Type in any number in place of 1025, and you have a metric for your group_concat_max_len.

SELECT
  CHAR_LENGTH(GROUP_CONCAT(REPEAT('0', 32768))) >= 32768 As group_concat_max_len_is_long_enough
FROM
  DUAL;
+-------------------------------------+
| group_concat_max_len_is_long_enough |
+-------------------------------------+
|                                   0 |
+-------------------------------------+

The above makes a computation with REPEAT. One can replace this with a big constant.

Some of these variables have different defaults as of MySQL 5.4. innodb_buffer_pool_size, for example, is 128M on 5.4. innodb_log_file_size, however, has changed back and forth, as far as I understand, and is down to 5M again. These settings are still the same on 5.5.

I wish to present some not-so-obvious parameters which, in my opinion, have poor defaults, for reasons I will explain.

• group_concat_max_len: This parameter limits the maximum text length of a GROUP_CONCAT concatenation result. It defaults to 1024. I think this is a very low value. I have been using GROUP_CONCAT more and more, recently, to solve otherwise difficult problems. And in most cases, 1024 was just too low, resulting in silent (Argh!) truncating of the result, thus returning incorrect results. It is interesting to learn that the maximum value for this parameter is limited by max_packet_size. I would suggest, then, that this parameter should be altogether removed, and have the max_packet_size limitation as the only limitation. Otherwise, I’d like it to have a very large default value, in the order of a few MB.
• wait_timeout: Here’s a parameter whose default value is over permissive. wait_timeout enjoys an 8 hour default. I usually go for 5-10 minutes. I don’t see a point in letting idle connections waste resources for 8 hours. Applications which hold up such connections should be aware that they’re doing something wrong, in the form of a forced disconnection. Connection pools work beautifully with low settings, and can themselves do keepalives, if they choose to.
• sql_mode: I’ve discussed this in length before. My opinion unchanged.
• open_files_limit: What with the fact connections, threads, table descriptors, table file descriptors (depending on how you use InnoDB), temporary file tables — all are files on unix-like systems, and considering this is an inexpensive payment, I think open_files_limit should default to a few thousands. Why risk the crash of “too many open files”?

No setting will ever be perfect for everyone, I know. But there are those parameters which you automatically set values for when you do a new install. These should be at focus and their defaults change.

It does not explain how to create replicating slaves. See How To Setup Replication for that. However, not all configuration options are well understood, and their roles in varying architectures can change.

Here are the settings for a basic Master/Slave(s) replication architecturee.

Essential

• log-bin: enable binary logs on the master. Replication is based on the master logging all modifying queries (INSERT/CREATE/ALTER/GRANT etc.), and the slaves being able to replicate them.
• server-id: each machine must have a unique server-id. A slave will not replay queries originating from a server with the same server-id as its own.
• GRANT: grant a user with REPLICATION SLAVE. The host list must include all replication slave hosts.
• expire-logs-days: automatically clean up master’s binary logs older than given value. By default, binary logs are never removed.

When working with Master/Slaves replication, one should be prepared to master failure and slave promotion to master. It may be desirable to identify a particular slave as primary candidate for promotion.

Just setting up the log-bin will yield with warnings in the MySQL’s error log. The binary logs are named, by default, after the host’s name. If that should change – MySQL will not be able to find the binary logs anymore (expecting a name which does previous logs did not use). It is therefore recommended to use:

log-bin=mychachine-bin

or

log-bin=mysql-bin

Essential/Optional

• log-bin: enable on a slave, so that in case it is promoted to master, the rest of the slaves can replicate using its binary logs. Enabling binary logging cannot be done on a live server: this parameter requires MySQL restart.
• GRANT: include the master’s host, so that when a slave promotes to master, the master can become a slave and continue replicating.
• log-slave-updates: together with log-bin, enable on slave so that master’s binary logs are propagated and logged by the slave. This is required if the slave takes the role of a master in a chained replication setup.
• expire-logs-days: set this flag on slave as well [tnx Sheeri].
• read-only: set on slave(s). Refuses any modifying query (INSERT, DELETE, ALTER, DROP etc.) for non-SUPER privileged users [tnx Ryan].
• sync-binlog: flush binary log to disk per transaction commit. Use this on master for safer replication; however note that increased I/O is expected [tnx Harrison].

Extra

• report-host, report-port: the host and port identifying the slave when looking at SHOW SLAVE HOSTS on master. Set this up on all hosts. See further discussion here.
• max-binlog-size: the maximum size for a binary log / relay log file, after which it is rotated.

Expert

• binlog-do-db, binlog-do-table, replicate-do-db, …: filter queries by either not writing them to binary log, or not reading them from the logs.

The reason I list the above as “Expert” is not because one must have a super-brain to set them up. That part is easy enough. But they lead to some dangerous situations, sometimes seemingly harmless. It takes great care to control the application and developers from creating those situations. See documentation here. See also discussion here and here.

• Using GROUP_CONCAT with a small group_concat_max_len setting? Your result will be silently truncated (make sure to check the warnings though).
• Calling CREATE TEMPORARY TABLE? You get silent commit.
• Issuing a ROLLBACK on non-transactional involved engines? Have a warning; no error.
• Using LOCK IN SHARE MODE on non transactional tables? Not a problem. Nothing reported.
• Adding a FOREIGN KEY on a MyISAM table? Good for you; no action actually taken.
• Inserting 300 to a TINYINT column in a relaxed sql_mode? Give me 255, I’ll silently drop the remaining 45. I owe you.

Warnings and errors

It would be nice to:

• Have an auto_propagate_warning_to_error server variable (global/session/both) which, well, does what it says.
• Have an i_am_really_not_a_dummy server variable which implies stricter checks for all the above and prevents you from doing with anything that may be problematic (or rolls back your transactions on your invalid actions).

Connectors may be nice enough to propagate warnings to errors – that’s good. But not enough: since data is already committed in MySQL.

If I understand correctly, and maybe it’s just a myth, it all relates to the times where MySQL had interest in a widespread adoption across the internet, in such way that it does not interfere too much with the users (hence leading to the common myth that “MySQL just works out of the box and does not require me to configure or understand anything”).

MySQL is a database system, and is now widespread, and is used by serious companies and products. It is time to stop play nice to everyone and provide with strict integrity — or, be nice to everyone, just allow me to specify what “nice” means for me.

Using binlog-do-db is dangerous to your replication. It means the master will not write to binary logs any statement not in the given database.

Ahem. Not exactly. It will not write to binary logs any statement which did not originate from the given database.

Which is why a customer, who was using Toad for MySQL as client interface to MySQL, and by default connected to the mysql schema, did not see his queries being replicated. In fact, he later on got replication errors. If you do:

USE test;
INSERT INTO world.City VALUES (...)

Then the statement is assumed to be in the test database, not in the world database.

Slightly better is using replicate-do-db on the slave machines. At least we allow the master to write everything. But still, for the same reasons, slaves may fail to repeat a perfectly valid query, just because it has been issued in the context of the wrong database. replicate-ignore-db is somewhat safer yet, but the trap is still there.

My advice is that replication should replicate everything. Make sure you and everyone else you work with understand the implications of binlog-do-db and replicate-do-db before implementing it.

Why?

I use yum and apt-get whenever I can and for almost all needs (sometimes preferring CPAN for Perl installations). But on a MySQL machine, I avoid doing so. The reason is either dependency hell or dependency mismatch.

Package managers are supposed to solve the dependency hell issue. But package managers will rarely have an up to date MySQL version.

I’ve had several experiences where a simple yum installation re-installed the MySQL version. I’ve had customers calling me up when, having installed something with yum, MySQL would not work anymore.

yum install package-which-depends-on-mysql-server will install MySQL server on your system if it hasn’t been installed with yum. Are you on CentOS 5.0? You’ll get MySQL 5.0.22. Oh, did you already have a RPM installation for MySQL 5.0.81? Sorry – it’s just been downgraded, plus it won’t work anymore since the error messages file has been changed since then.

Don’t press ‘Y‘ too soon!

Things are slightly better with apt-get. I’ve encountered less situations where mysql-server was on the dependency list. Many times it’s just the libmysqlclient package or the mysql-common one.

But wait! Did you install mysql-common? Bonus! You get the elusive /etc/mysql/my.cnf file created, and there goes your server configuration. Future spawns of the MySQL server / clients will read from the wrong configuration file, and will probably fail to load.

Not to mention neither will help you out with multiple instances installation.

My argument

A sys admin recently argued with me that it was wrong of me to have the entire machine set up with yum, but have MySQL installed with binary tarball. He argued that it broke the entire setup. I expressed my opinion: on a MySQL dedicated server, MySQL gets to be prioritized. It’s special. It is the reason for the existence of the machine. I would imagine that same would hold for Apache on an Apache dedicated machine, for Sendmail on a Sendmail dedicated machine, etc. As a DBA, I want to have best control of the MySQL installation; I want to be able to upgrade minor versions quickly: I often find newer versions to solve bugs I was concerned with; I want to be able to install multiple instances; I want to be able to downgrade without having to remove and uninstall the previous version.

I want to have control. World domination aside, that is.

mycheckpoint is a different kind of monitoring tool. It leaves the power in the user’s hand. It’s power is not with script-based calculations of recorded data. It’s with the creation of a view hierarchy, which allows the user to access computed metrics directly.

mycheckpoint is needed first, to deploy a monitoring schema. It may be needed next, so as to INSERT recorded data (GLOBAL STATUS, GLOBAL VARIABLES, MASTER STATUS, SLAVE STATUS) — but this is just a simple INSERT; anyone can do that, even another monitoring tool.

It is then that you do not need it anymore: everything is laid at your fingertips. Consider:

SELECT innodb_read_hit_percent, DML FROM sv_report_chart_hour;

mycheckpoint provides the views which take raw data (just innodb_buffer_pool_read_requests, com_select, innodb_buffer_pool_size, table_open_cache, seconds_behind_master etc.) and generate Google Charts URLs, HTML reports, human readable reports, or otherwise easily accessible data.

Data is provided in different time resolutions:

• Per sampling
• Per hour aggregated data
• Per day aggregated data

It is thus easy to get a fine grained or a daily overview of your status. In fact, the SQL-generated HTML report lays them all together.

[Read more on generating Google Charts and HTML reports]

It is more about data accessibility

Charts are cool to look at, but they are not useful for detailed analysis. The user is free to ask anything of the supporting views:

I want to see the average number of SELECT queries per second in the last 5 hours:

mysql> SELECT ts, com_select_psec FROM sv_hour ORDER BY id DESC LIMIT 5;
+---------------------+-----------------+
| ts                  | com_select_psec |
+---------------------+-----------------+
| 2009-11-09 11:00:00 |          294.17 |
| 2009-11-09 10:00:00 |          198.37 |
| 2009-11-09 09:00:00 |          151.29 |
| 2009-11-09 08:00:00 |           90.06 |
| 2009-11-09 07:00:00 |           82.98 |
+---------------------+-----------------+

Hmm. Seems like too many SELECTs in the last hour.

Unrelated, is the InnoDB buffer pool being utilized well?

mysql> SELECT ts, innodb_buffer_pool_used_percent, innodb_read_hit_percent
       FROM sv_report_sample
       ORDER BY id DESC LIMIT 5;
+---------------------+---------------------------------+-------------------------+
| ts                  | innodb_buffer_pool_used_percent | innodb_read_hit_percent |
+---------------------+---------------------------------+-------------------------+
| 2009-11-09 12:35:01 |                           100.0 |                   99.93 |
| 2009-11-09 12:30:01 |                           100.0 |                   99.89 |
| 2009-11-09 12:25:01 |                           100.0 |                   99.60 |
| 2009-11-09 12:20:01 |                           100.0 |                   99.14 |
| 2009-11-09 12:15:01 |                           100.0 |                   98.99 |
+---------------------+---------------------------------+-------------------------+

Apparently, innodb_buffer_pool_size could use some more memory.

When did we have excessive amount of writes?

mysql> SELECT ts, com_insert_psec
       FROM sv_hour
       WHERE com_insert_psec > (SELECT 2*AVG(com_insert_psec) FROM sv_hour);
+---------------------+-----------------+
| ts                  | com_insert_psec |
+---------------------+-----------------+
| 2009-10-27 00:00:00 |          133.66 |
| 2009-10-28 00:00:00 |          121.79 |
| 2009-10-29 00:00:00 |          138.88 |
| 2009-10-30 00:00:00 |          120.79 |
| 2009-10-31 00:00:00 |          131.78 |
+---------------------+-----------------+

Something is going on on those midnights!

[Read more on querying for data]

Human reports

But while we’re at it: it’s nice to let the user the ability to ask around; but why not provide with some niceties? Special views aggregate monitored data to present human readable reports:

SELECT report FROM sv_report_human_hour ORDER BY id DESC LIMIT 1,1 \G

Report period: 2009-11-08 14:00:00 to 2009-11-08 15:00:00. Period is 60 minutes (1.00 hours)
Uptime: 100.0% (Up: 285 days, 07:17:28 hours)

InnoDB:
    innodb_buffer_pool_size: 4718592000 bytes (4500.0MB). Used: 100.0%
    Read hit: 99.75%
    Disk I/O: 83.00 reads/sec  20.33 flushes/sec
    Estimated log written per hour: 797.0MB
    Locks: 0.32/sec  current: 0

MyISAM key cache:
    key_buffer_size: 33554432 bytes (32.0MB). Used: 18.3%
    Read hit: 99.7%  Write hit: 100.0%

DML:
    SELECT:  149.88/sec  34.1%
    INSERT:  55.84/sec  12.7%
    UPDATE:  17.55/sec  4.0%
    DELETE:  20.68/sec  4.7%
    REPLACE: 0.00/sec  0.0%
    SET:     170.05/sec  38.7%
    COMMIT:  0.02/sec  0.0%
    slow:    2.28/sec  0.5% (slow time: 2sec)

Selects:
    Full scan: 8.37/sec  5.6%
    Full join: 0.00/sec  0.0%
    Range:     40.45/sec  27.0%
    Sort merge passes: 0.00/sec

Locks:
    Table locks waited:  0.00/sec  0.0%

Tables:
    Table cache: 2048. Used: 26.5%
    Opened tables: 0.00/sec

Temp tables:
    Max tmp table size:  67108864 bytes (64.0MB)
    Max heap table size: 67108864 bytes (64.0MB)
    Created:             7.15/sec
    Created disk tables: 0.51/sec  7.1%

Connections:
    Max connections: 200. Max used: 245  122.5%
    Connections: 3.31/sec
    Aborted:     0.07/sec  2.1%

Threads:
    Thread cache: 32. Used: 50.0%
    Created: 0.06/sec

Replication:
    Master status file number: 1494, position: 404951764
    Relay log space limit: 10737418240, used: N/A  (N/A%)
    Seconds behind master: N/A
    Estimated time for slave to catch up: N/A seconds (N/A days, N/A hours)  ETA: N/A

The above is a SQL-generated report. The view’s CREATE statement is ugly, trust me! But the user needs not be aware of this — all is generated behind the scenes. Since it is SQL-generated, the report is not actually stored anywhere; and one can generate reports for as long as data exists. A three months old data can still be evaluated and used to produce a fresh report.

The above report resembles the ever-so-useful mysqlreport by Daniel Nichter. I have drawn many ideas from this tool.

[Read more on generating human readable reports]

Tracking change of parameters

Since mycheckpoint records server variables, it’s easy enough to detect a change in variable. Did you dynamically change a variable and forgot to update my.cnf? Were you baffled when the server restarted and everything started behaving differently? Just ask away:

mysql> SELECT * FROM sv_param_change;
+---------------------+-----------------+-----------+-----------+
| ts                  | variable_name   | old_value | new_value |
+---------------------+-----------------+-----------+-----------+
| 2009-11-04 13:00:01 | max_connections |       500 |       200 |
+---------------------+-----------------+-----------+-----------+

Doh! That’s how we got 122.5% max used connections!

[Read more on detecting parameters change]

Additional notes

Just recently, a somewhat similar project, sar-sql was announced by Gerry Narvaja (Ex-Pythian). When sar-sql (formerly MySAR) was announced, my own code and ideas were at late stages. I’ve pondered about this, and have decided to go on with a separate project. While both make use of the same ideas, the implementation is quite different.

With proper setup, mycheckpoint can be used as an add-on to other monitoring tools. I currently have no plans for doing that, but time will tell.

I believe the ease of access to monitored data is a compelling reason to try out mycheckpoint. Please visit the mycheckpoint home page, read through the documentation, and take some downloads with you!

As always, community feedback is welcome. Feel free to throw in valueable feedback, bug reports or even a couple of tomatoes!

mycheckpoint is released under the BSD license.

What I usually find in schemes are columns which are either utf8 or latin1. The utf8 columns being those which need to contain multilingual characters (user names, addresses, articles etc.), and latin1 column being all the rest (passwords, digests, email addresses, hard-coded values etc.)

I find latin1 to be improper for such purposes and suggest that ascii be used instead. The reason being that latin1 implies a European text (with swedish collation). It is unclear for an outsider, when finding a latin1 column, whether it should actually contain West European characters, or is it just being used for ascii text, utilizing the fact that a character in latin1 only requires 1 byte of storage.

Well, this is what the ascii character set is for. When I see an ascii column, I know for sure no West European characters are allowed; just the plain old a-zA-Z0-9 etc. It is clearer from the schema’s definition what the stored values should be.

A note to MySQL

It’s been long since the Swedish roots of the company have dictated defaults. New instances should default to either ascii or  utf8 (the latter being the most common and space efficient unicode protocol): character sets that are locale-neutral. Really, how many people realize that when they ORDER BY a text column, rows are sorted according to Swedish dictionary ordering?

1. Throw everything in one big file (optionally split).
2. Have one file per table.

I will discuss the advantages and disadvantages of the two options, and will strive to convince that innodb_file_per_table is preferable.

A single tablespace

Having everything in one big file means all tables and indexes, from all schemes, are ‘mixed’ together in that file.

This allows for the following nice property: free space can be shared between different tables and different schemes. Thus, if I purge many rows from my log table, the now unused space can be occupied by new rows of any other table.

This same nice property also translates to a not so nice one: data can be greatly fragmented across the tablespace.

An annoying property of InnoDB’s tablespaces is that they never shrink. So after purging those rows from the log table, the tablespace file (usually ibdata1) still keeps the same storage. It does not release storage to the file system.

I’ve seen more than once how certain tables are left unwatched, growing until disk space reaches 90% and SMS notifications start beeping all around.

There’s little to do in this case. Well, one can always purge the rows. Sure, the space would be reused by InnoDB. But having a file which consumes some 80-90% of disk space is a performance catastrophe. It means the disk needle needs to move large distances. Overall disk performance runs very low.

The best way to solve this is to setup a new slave (after purging of the rows), and dump the data into that slave.

InnoDB Hot Backup

The funny thing is, the ibbackup utility will copy the tablespace file as it is. If it was 120GB, of which only 30GB are used, you still get a 120GB backed up and restored.

mysqldump, mk-parallel-dump

mysqldump would be your best choice if you only had the original machine to work with. Assuming you’re only using InnoDB, a dump with –single-transaction will do the job. Or you can utilize mk-parallel-dump to speed things up (depending on your dump method and accessibility needs, mind the locking).

innodb_file_per_table

With this parameter set, a .ibd file is created per table. What we get is this:

• Tablespace is not shared among different tables, and certainly not among different schemes.
• Each file is considered a tablespace of its own.
• Again, tablespace never reduces in size.
• It is possible to regain space per tablespace.

Wait. The last two seem conflicting, don’t they? Let’s explain.

In our log table example, we purge many rows (up to 90GB of data is removed). The .ibd file does not shrink. But we can do:

ALTER TABLE log ENGINE=InnoDB

What will happen is that a new, temporary file is created, into which the table is rebuilt. Only existing data is added to the new table. Once comlete, the original table is removed, and the new table renamed as the original table.

Sure, this takes a long time, during which the table is completely locked: no writes and no reads allowed. But still – it allows us to regain disk space.

With the new InnoDB plugin, disk space is also regained when execuing a TRUNCATE TABLE log statement.

Fragmentation is not as bad as in a single tablespace: the data is limited within the boundaries of a smaller file.

Monitoring

One other nice thing about innodb_file_per_table is that it is possible to monitor table size on the file system level. You don’t need access to MySQL, to use SHOW TABLE STATUS or to query the INFORMATION_SCHEMA. You can just look up the top 10 largest files under your MySQL data directory (and subdirectories), and monitor their size. You can see which table grows fastest.

Backup

Last, it is not yet possible to backup single InnoDB tables by copying the .ibd files. But hopefully work will be done in this direction.