How much can a single consultant observe? I agree Oracle should not go to PR based on my experience. But I find that:

• More companies are now familiar with InnoDB than there used to.
• More companies are interested in migration to InnoDB than there used to.
• More companies feel such migration to be safe.
• More companies start up with an InnoDB based solution than with a MyISAM based solution.

This is the way I see it. No doubt, the Oracle/Sun deal made its impact. The fact that InnoDB is no longer a 3rd party; the fact Oracle invests in InnoDB and no other engine (Falcon is down, no real development on MyISAM); the fact InnoDB is to be the default engine: all these put companies at ease with migration.

I am happy with this change. I believe for most installations InnoDB provides with a clear advantage over MyISAM (though MyISAM has its uses), and this makes for more robust, correct and manageable MySQL instances; the kind that make a DBA’s life easier and quieter. And it is easier to make customers see the advantages.

I am not inclined to say “You should migrate your entire database to InnoDB”. I don’t do that a lot. But recently, more customers approach and say “We were thinking about migrating our entire database to InnoDB, what do you think?”. What a change of approach.

And, yes: there are still a lot of companies using MyISAM based databases, who still live happily.

I’m just still encountering companies who use MyISAM as their storage engine and are unaware that their nightly backup actually blocks their application, basically rendering their product unavailable for long minutes to hours on a nightly basis.

So this is posted as a warning for those who were not aware of this fact.

There is no hot (non blocking) backup for MyISAM. Closest would be file system snapshot, but even this requires flushing of tables, which may take a while to complete. If you must have a hot backup, then either use replication – and take the risk of the slave not being in complete sync with the master – or use another storage engine, i.e. InnoDB.

Database users were experiencing low responsiveness. DBAs were seeing locks occurring on seemingly normal tables. In particular, looking at Innotop, it seemed that INSERTs were causing the locks.

In both cases, tables were InnoDB. In both cases, there was a PRIMARY KEY on the combination of all 5 columns. And in both cases, there was no clear explanation as for why the PRIMARY KEY was chosen as such.

Choosing a proper PRIMARY KEY

Especially with InnoDB, which uses clustered index structure, the PRIMARY KEY is of particular importance. Besides the fact that a bloated PRIMARY KEY bloats the entire clustered index and secondary keys (see: The depth of an index: primer), it is also a source for locks. It’s true that any UNIQUE KEY can serve as a PRIMARY KEY. But not all such keys are good candidates.

Reducing the locks

In both described cases, the solution was to add an AUTO_INCREMENT column to serve as the PRIMARY KEY, and have that 5 column combination under a secondary UNIQUE KEY. The impact was immediate: no further locks on that table were detected, and query responsiveness turned very high.

TRUNCATE on tables with no FOREIGN KEYs should act fast: it translate to dropping the table and creating a new one (and it all depends on the MySQL version, see the manual).

What’s faster than TRUNCATE, then? If you don’t have triggers nor FOREIGN KEYs, a RENAME TABLE can come to the rescue. Instead of:

TRUNCATE log_table

Do:

CREATE TABLE log_table_new LIKE log_table;
RENAME TABLE log_table TO log_table_old, log_table_new TO log_table;
DROP TABLE log_table_old;

I found this to work well for me. Do note that AUTO_INCREMENT values can be tricky here: the “new” table is created with an AUTO_INCREMENT value which is immediately taken in the “working” table. If you care about not using same AUTO_INCREMENT values, you can:

ALTER TABLE log_table_new AUTO_INCREMENT=some high enough value;

Just before renaming.

I do not have a good explanation as for why the RENAME TABLE succeeds to respond faster than TRUNCATE.

InnoDB performance

Taking a look at one of the most important InnoDB metrics: the read hit ratio (we could get the same graph by looking at the HTML report):

SELECT innodb_read_hit_percent FROM sv_report_chart_sample \G
*************************** 1. row ***************************
innodb_read_hit_percent: http://chart.apis.google.com/chart?cht=lc&chs=400x200&chts=303030,12&chtt=Nov+10,+11:40++-++Nov+11,+08:55+(0+days,+21+hours)&chdl=innodb_read_hit_percent&chdlp=b&chco=ff8c00&chd=s:400664366P6674y7176677677u467773y64ux166666764366646y616666666666644444434444s6u4S331444404433341334433646777666666074736777r1777767764776666F667777617777777777777777yaRi776776mlf667676xgx776766rou67767777u37797777x76676776u6A737464y67467761777666643u66446&chxt=x,y&chxr=1,99.60,100.00&chxl=0:||Nov+10,+15:55|Nov+10,+20:10|Nov+11,+00:25|Nov+11,+04:40|&chxs=0,505050,10

We see that read hit is usually high, but occasionally drops low, down to 99.7, or even 99.6. But it seems like most of the time we are above 99.95% read hit ratio. It’s hard to tell about 99.98%.

Can we know for sure?

We can stress our eyes, yet be certain of little. It’s best if we just query for the metrics! mycheckpoint provides with all data, accessible by simple SQL queries:

SELECT SUM(innodb_read_hit_percent > 99.95)/count(*)
  FROM sv_report_sample;
+-----------------------------------------------+
| SUM(innodb_read_hit_percent > 99.95)/count(*) |
+-----------------------------------------------+
|                                        0.7844 |
+-----------------------------------------------+

Yes, most of the time we’re above 99.95% read hit ratio: but not too often!

I’m more interested in seeing how much time my server’s above 99.98% read hit:

SELECT SUM(innodb_read_hit_percent > 99.98)/count(*)
  FROM sv_report_sample;
+-----------------------------------------------+
| SUM(innodb_read_hit_percent > 99.98)/count(*) |
+-----------------------------------------------+
|                                        0.3554 |
+-----------------------------------------------+

We can see the server only has 99.98% read hit percent 35% of the time. Need to work on that!

Disk activity

Lower read hit percent means higher number of disk reads; that much is obvious. The first two following graphs present this obvious connection. But the third graph tells us another fact: with increased disk I/O, we can expect more (and longer) locks.

Again, this should be very intuitive, when thinking about it this way. The problem sometimes arises when we try to analyze it the other way round: “Hey! InnoDB has a lot of locks! What are we going to do about it?”. Many times, people will look for answers in their transactions, their Isolation Level, their LOCK IN SHARE MODE clauses. But the simple answer can be: “There’s a lot of I/O, so everything has to wait; therefore we increase the probability for locks; therefore there’s more locks”.

The answer, then, is to reduce I/O. The usual stuff: slow queries; indexing; … and, yes, perhaps transactions or tuning.

The charts below make it quite clear that we have an issue of excessive reads -> less read hit -> increased I/O -> more locks.

By the way, the above resulted from the fact that, due to a problematic query, all slave stopped replicating. Slaves participated in read-balancing, so when they went stale, all reads were directed at the master (the monitored node).

You have the metrics at your disposal

Looking at the following chart:

It appears that there’s no slow queries. But this may be misleading: perhaps there’s just a little, that don’t show due to the chart’s large scale?

One could argue that this is the chart’s fault. Perhaps there should be a distinct chart for “slow queries percent”. Perhaps I’ll add one. But we can’t have special charts for everything. It’s would be too tiresome to look at hundreds of charts.

Anyway, my point is: let’s verify just how many slow queries we have:

SELECT slow_queries_psec FROM sv_hour ORDER BY id DESC;
+-------------------+
| slow_queries_psec |
+-------------------+
|              3.05 |
|              3.83 |
|              4.39 |
|              4.03 |
|              3.86 |
|              3.56 |
|              3.73 |
|              3.79 |
|              3.58 |
|              3.55 |
...
+-------------------+

So, between 3 and 4 slow queries per second. It doesn’t look too good in this light. Checking on the percentage of slow queries (of total questions):

SELECT ROUND(100*slow_queries_diff/questions_diff, 1) AS slow_queries_percent
  FROM sv_hour ORDER BY id DESC LIMIT 10;

Or, since the above calculation is pre-defined in the reports tables:

SELECT slow_queries_percent FROM sv_report_hour_recent;
+----------------------+
| slow_queries_percent |
+----------------------+
|                  0.8 |
|                  1.0 |
|                  1.2 |
|                  1.2 |
|                  1.1 |
|                  1.0 |
|                  1.1 |
|                  1.1 |
|                  1.0 |
...
+----------------------+

Accessible data

This is what I’ve been trying to achieve with mycheckpoint. As a DBA, consultant and SQL geek I find that direct SQL access works best for me. It’s like loving command line interface over GUI tools. Direct SQL gives you so much more control and information.

Charting is important, since it’s easy to watch and get first impressions, or find extreme changes. But beware of relying on charts all the time. Scale issues, misleading human interpretation, technology limitations – all these make charts inaccurate.

mycheckpoint allows for both methods, and, I believe, intuitively so.

</propaganda>

Recap: this information can be useful if you’re looking for a good innodb_log_file_size value, such that will not pose too much I/O (smaller values will make for more frequent flushes), not will make for a too long recovery time (larger values mean more transactions to recover upon crash).

It is assumed that the 60 seconds period represents an average system load, not some activity spike period. Edit the sleep time and factors as you will to sample longer or shorter periods.

SELECT
  innodb_os_log_written_per_minute*60
    AS estimated_innodb_os_log_written_per_hour,
  CONCAT(ROUND(innodb_os_log_written_per_minute*60/1024/1024, 1), 'MB')
    AS estimated_innodb_os_log_written_per_hour_mb
FROM
  (SELECT SUM(value) AS innodb_os_log_written_per_minute FROM (
    SELECT -VARIABLE_VALUE AS value
      FROM INFORMATION_SCHEMA.GLOBAL_STATUS
      WHERE VARIABLE_NAME = 'innodb_os_log_written'
    UNION ALL
    SELECT SLEEP(60)
      FROM DUAL
    UNION ALL
    SELECT VARIABLE_VALUE
      FROM INFORMATION_SCHEMA.GLOBAL_STATUS
      WHERE VARIABLE_NAME = 'innodb_os_log_written'
  ) s1
) s2
;

Sample output:

+------------------------------------------+---------------------------------------------+
| estimated_innodb_os_log_written_per_hour | estimated_innodb_os_log_written_per_hour_mb |
+------------------------------------------+---------------------------------------------+
|                                584171520 | 557.1MB                                     |
+------------------------------------------+---------------------------------------------+

In all those case, I needed to justify the move to “new technology”. The questions “Is it GA? Is it stable?” are being asked a lot. Well, just a few days ago the MySQL 5.1 distribution started shipping with InnoDB plugin 1.0.4. That gives some weight to the stability question when facing a doubtful customer.

But I realized that wasn’t the point.

Before InnoDB plugin was first announced, little was going on with InnoDB. There were concerns about the slow/nonexistent progress on this important storage engine, essentially the heart of MySQL. Then the plugin was announced, and everyone went happy.

The point being, since then I only saw (or was exposed to, at least) progress on the plugin. The way I understand it, the plugin is the main (and only?) focus of development. And this is the significant thing to consider: if you’re keeping to “old InnoDB”, fine – but it won’t get you much farther; you’re unlikely to see great performance improvements (will 5.4 make a change? An ongoing improvement to InnoDB?). It may eventually become stale.

Converting to InnoDB plugin means you’re working with the technology at focus. It’s being tested, benchmarked, forked, improved, talked about, explained. I find this to be a major motive.

So, long live InnoDB Plugin! (At least till next year, that is, when we may all find ourselves migrating to PBXT)

I’m going to leave out “performance” for the reason that grander forces have benchmarked and written about it.

Compression

Using the new Barracuda table format, table data can be compressed. Compression depends on the type of data you have in your table, and in KEY_BLOCK_SIZE. I have found tables with lots of textual data to compress well, to about 25% volume (that is, reduction of 75%), and strictly integer-typed tables (like an a-2-b connecting table) to compress poorly.

I have seen an InnoDB 50GB database shrink into some 12GB only. Wow! That meant a server which only had RAID 1 two 72GB disks, and which was dangerously filled up with disk space, could now accommodate the database, a backup, and then some!

Compression does not only occur on disk: when pages are loaded to memory, they are loaded compressed. This means the same innodb_buffer_pool_size you had, now holds a lot more data.

There is no compression for the undo buffer.

INFORMATION_SCHEMA

There’s lot’s of useful data in the new INFORMATION_SCHEMA tables. Very interesting questions can be answered:

• How many transactions are open right now?
• What’s the state of each transaction?
• What queries are being run right now?
• Are any transaction waiting on locks? Which locks? Held by which transactions?
• What kind of locks are currently held? On which tables?
• How much time has been spent on compression/uncompress?
• How many compression operations have occurred?
• How many pages are allocated? How many are free?

Fast index creation

Just to be clear, this is not non-blocking index creation. It’s just fast. What’s fast? Well, slow index creation is what we’re used to: to add an index we ALTER TABLE, thereby creating a new, temporary tablespace, into which the entire table’s data is copied. With fast index creation, table data is left untouched. The new index is created by scanning the primary key and to-be indexed values. Thus, it requires less I/O operations.

Dropping an index is an instantaneous operation.

1. Throw everything in one big file (optionally split).
2. Have one file per table.

I will discuss the advantages and disadvantages of the two options, and will strive to convince that innodb_file_per_table is preferable.

A single tablespace

Having everything in one big file means all tables and indexes, from all schemes, are ‘mixed’ together in that file.

This allows for the following nice property: free space can be shared between different tables and different schemes. Thus, if I purge many rows from my log table, the now unused space can be occupied by new rows of any other table.

This same nice property also translates to a not so nice one: data can be greatly fragmented across the tablespace.

An annoying property of InnoDB’s tablespaces is that they never shrink. So after purging those rows from the log table, the tablespace file (usually ibdata1) still keeps the same storage. It does not release storage to the file system.

I’ve seen more than once how certain tables are left unwatched, growing until disk space reaches 90% and SMS notifications start beeping all around.

There’s little to do in this case. Well, one can always purge the rows. Sure, the space would be reused by InnoDB. But having a file which consumes some 80-90% of disk space is a performance catastrophe. It means the disk needle needs to move large distances. Overall disk performance runs very low.

The best way to solve this is to setup a new slave (after purging of the rows), and dump the data into that slave.

InnoDB Hot Backup

The funny thing is, the ibbackup utility will copy the tablespace file as it is. If it was 120GB, of which only 30GB are used, you still get a 120GB backed up and restored.

mysqldump, mk-parallel-dump

mysqldump would be your best choice if you only had the original machine to work with. Assuming you’re only using InnoDB, a dump with –single-transaction will do the job. Or you can utilize mk-parallel-dump to speed things up (depending on your dump method and accessibility needs, mind the locking).

innodb_file_per_table

With this parameter set, a .ibd file is created per table. What we get is this:

• Tablespace is not shared among different tables, and certainly not among different schemes.
• Each file is considered a tablespace of its own.
• Again, tablespace never reduces in size.
• It is possible to regain space per tablespace.

Wait. The last two seem conflicting, don’t they? Let’s explain.

In our log table example, we purge many rows (up to 90GB of data is removed). The .ibd file does not shrink. But we can do:

ALTER TABLE log ENGINE=InnoDB

What will happen is that a new, temporary file is created, into which the table is rebuilt. Only existing data is added to the new table. Once comlete, the original table is removed, and the new table renamed as the original table.

Sure, this takes a long time, during which the table is completely locked: no writes and no reads allowed. But still – it allows us to regain disk space.

With the new InnoDB plugin, disk space is also regained when execuing a TRUNCATE TABLE log statement.

Fragmentation is not as bad as in a single tablespace: the data is limited within the boundaries of a smaller file.

Monitoring

One other nice thing about innodb_file_per_table is that it is possible to monitor table size on the file system level. You don’t need access to MySQL, to use SHOW TABLE STATUS or to query the INFORMATION_SCHEMA. You can just look up the top 10 largest files under your MySQL data directory (and subdirectories), and monitor their size. You can see which table grows fastest.

Backup

Last, it is not yet possible to backup single InnoDB tables by copying the .ibd files. But hopefully work will be done in this direction.

In both these structures, the depth of the index is an important factor. When looking for an indexed row, a search is made on the index, from root to leaves.

Assuming the index is not in memory, the depth of the index represents the minimal cost (in I/O operation) for an index based lookup. Of course, most of the time we expect large portions of the indexes to be cached in memory. Even so, the depth of the index is an important factor. The deeper the index is, the worse it performs: there are simply more lookups on index nodes.

What affects the depth of an index?

There are quite a few structural issues, but it boils down to two important factors:

1. The number of rows in the table: obviously, more rows leads to larger index, larger indexes grow in depth.
2. The size of the indexed column(s). An index on an INT column can be expected to be shallower than an index on a CHAR(32) column (on a very small number of rows they may have the same depth, so we’ll assume a large number of rows).

Of course, these two factors also affect the total size of the index, hence its disk usage, but I wish to concentrate on the index depth.

Let’s emphasize the second factor. It is best to index shorter columns, if that is possible. It is the reason behind using an index on a VARCHAR’s prefix (e.g. KEY(email_address(16)). It is also a reason to use INT, instead of BIGINT columns for your primary key, when BIGINT is not required.

The larger the indexed data type is (or the total size of data types for all columns in a combined index), the less values that can fit in an index node. The less values in a node, the more node splits occur; the more nodes are required to build the index. The less values in the node, the less wide the index tree is. The less wide an index tree is, and the more nodes it has – the deeper it gets.

So bigger data types lead to deeper trees. Deeper trees lead to more IO operations on lookup.

InnoDB

On InnoDB there’s another issue: all tables are clustered by primary key. Any access to table data requires diving into, or traversing the primary key tree.

On InnoDB, a secondary index (any index which is not the primary key) does not lead to table data. Instead, the “data” in the leaf nodes of a secondary index – are the primary key values.

And so, when looking up a value on an InnoDB table using a secondary key, we first search the secondary key to retrieve the primary key value, then go to the primary key tree to retrieve the data.

This means two index lookups, one of which is always the primary key.

On InnoDB, it is therefore in particular important to keep the primary key small. Have small data types. Prefer an SMALLINT to INT, if possible. Prefer an INT to BIGINT, if possible. Prefer an integer value over some VARCHAR text.

With long data types used in an InnoDB primary key, not only is the primary key index bloated (deep), but also every other index gets to be bloated, as the leaf values in all other indexes are those same long data types.

MyISAM

MyISAM does not use clustered trees, hence the primary key is just a regular unique key. All indexes are created equal and an index lookup only consists of a single index search. Therefore, two indexes do no affect one another, with the exception that they are competing on the same key cache.