A particular direction I’m taking is to filter queries by their type of actions. For example, the tool (oak-hook-general-log) can be instructed to only stream out those queries which involve creation of a temporary table; or those which cause for a filesort, or full index scan, etc.

This is done by evaluating of query execution plans on the fly. I suspect the MySQL query analyzer roughly does the same (as a small part of what it does).

It’s almost nothing one cannot do with sed/awk. However, I bumped into a couple of problems:

1. The general log (and the mysql.general_log table, in  particular) does not indicate the particular database one is using for the query. Since slow log does indicate this data, I filed a bug on this. I currently solve this by crossing connection id with the process list, where the current database is listed. It’s shaky, but mostly works.
2. Just realized: there’s no DB info in the EXPLAIN output! It’s weird, since I’ve been EXPLAINing queries for years now. But I’ve always had the advantage of knowing the schema used: either because I was manually executing the query on a known schema, or mk-query-digest was kind enough to let me know.

For example, look at the following imaginary query, involving both the world and sakila databases:

mysql> use test;
Database changed
mysql> EXPLAIN SELECT * FROM world.Country JOIN sakila.city WHERE Country.Capital = city.city_id;
+----+-------------+---------+--------+---------------+---------+---------+-----------------------+------+-------------+
| id | select_type | table   | type   | possible_keys | key     | key_len | ref                   | rows | Extra       |
+----+-------------+---------+--------+---------------+---------+---------+-----------------------+------+-------------+
|  1 | SIMPLE      | Country | ALL    | NULL          | NULL    | NULL    | NULL                  |  239 |             |
|  1 | SIMPLE      | city    | eq_ref | PRIMARY       | PRIMARY | 2       | world.Country.Capital |    1 | Using where |
+----+-------------+---------+--------+---------------+---------+---------+-----------------------+------+-------------+
2 rows in set (0.00 sec)

The query is imaginary, since the tables don’t actually have anything in common. But look at the EXPLAIN result: can you tell where city came from? Country can somehow be parsed from the ref column, but no help on city.

Moreover, table aliases show on the EXPLAIN plan (which is good), but with no reference to the original table.

So, is it back to parsing of the SQL query? I’m lazy reluctant to do that. It’s error prone, and one needs to implement, or use, a good parser, which also accepts MySQL dialect. Haven’t looked into this yet.

I’m currently at a standstill with regard to automated query execution plan evaluation where database cannot be determined.

The tool, oak-hook-general-log, will hook up to a MySQL (>= 5.1) server, and stream the general log into standard output. It looks like this:

bash$ python src/oak/oak-hook-general-log.py --socket=/tmp/mysql.sock --user=root
2010-03-21 10:18:42     root[root] @ localhost []       79      1       Query   SELECT COUNT(*) FROM City
2010-03-21 10:18:48     root[root] @ localhost []       79      1       Query   DELETE FROM City WHERE id=1000
2010-03-21 10:18:54     root[root] @ localhost []       79      1       Query   SHOW PROCESSLIST
2010-03-21 10:19:06     root[root] @ localhost []       79      1       Quit
2010-03-21 10:19:07     root[root] @ localhost []       93      1       Connect root@localhost on
2010-03-21 10:19:07     root[root] @ localhost []       93      1       Query   select @@version_comment limit 1
2010-03-21 10:22:33     root[root] @ localhost []       93      1       Query   SELECT City.Name, Country.Name FROM Country JOIN City ON Country.Capit
2010-03-21 10:22:58     root[root] @ localhost []       93      1       Quit

Since output is written to stdout, one can further:

bash$ python src/oak/oak-hook-general-log.py --socket=/tmp/mysql.sock --user=root | grep Connect
bash$ python src/oak/oak-hook-general-log.py --socket=/tmp/mysql.sock --user=root | grep webuser@webhost

What the tool does is to enable table logs, and periodically rotate the mysql.general_log table, read and dump its content.

The tool:

• Stores and restores the original log state (general log enabled/disabled, log output).
• Disables printing of its own queries to the general log.
• Automatically times out (timeout configurable) so as not to enter a situation where the general log is forgotten to be turned on.
• Can discard pre-existing data on the mysql.general_log table.
• Will cleanup the mysql.slow_log table, if it wasn’t originally used (turning on table logs applies to both general log and slow log).

What would you have the tool do further? Should it provide filtering, or should we just use grep/sed/awk for that? Any internal aggregation of data?

I would love to hear your thoughts. Meanwhile, view or grab the python script file.

I apologize for not commenting on the post itself, I do not hold a Facebook account. Anyway this is a long write, so it may as well deserve a post of its own.

Some of the work Mark is describing already exists under openark kit‘s oak-online-alter-table. Allow me to explain what I have gained there, and how the issue can be further pursued. There is relevance to Mark’s suggestion.

oak-online-alter-table uses a combination of locks, chunks and triggers to achieve an almost non-blocking ALTER TABLE effect. I had a very short opportunity to speak with Mark on last year’s conference, in between bites. Mark stated that anything involving triggers was irrelevant in his case.

The triggers are a pain, but I believe a few other insights from oak-online-alter-table can be of interest.

The first attempt

My first attempt with the script assumed:

• Table has an AUTO_INCREMENT PRIMARY KEY column
• New rows always gain ascending PRIMARY KEY values
• PRIMARY KEY never changes for an existing row
• PRIMARY KEY values are never reused
• Rows may be deleted at will
• No triggers exist on the table
• No FOREIGN KEYs exist on the table.

So the idea was: when one wants to do an ALTER TABLE:

1. Create a ghost table with the new structure.
2. Read the minimum and maximum PK values.
3. Create AFTER INSERT, AFTER UPDATE, AFTER DELETE triggers on the original table. These triggers will propagate the changes onto the ghost table.
4. Working out slowly, and in small chunks, copy rows within recorded min-max values range into the ghost table. The interesting part is where the script makes sure there’s no contradiction between these actions and those of the triggers, (whichever came first!). This is largely solved using INSERT IGNORE and REPLACE INTO in the proper context.
5. Working out slowly and in chunks again, we remove rows from the ghost table, which are no longer existent in the original table.
6. Once all chunking is complete, RENAME original table to *_old, and ghost table in place of the original table.

Steps 4 & 5 are similar in concept to transactional recovery through redo logs and undo logs.

The next attempt

Next phase removed the AUTO_INCREMENT requirement, as well as the “no reuse of PK”. In fact, the only remaining constraints were:

• There is some UNIQUE KEY on the table which is unaffected by the ALTER operation
• No triggers exist on the table
• No FOREIGN KEYs exist on the table.

The steps are in general very similar to those listed previously, only now a more elaborate chunking method is used with possible non-integer, possible multi-column chunking algorithm. Also, the triggers take care of changes in UNIQUE KEY values themselves.

mk-schema-change?

Have a look at the wiki pages for OnlineAlterTable*. There is some discussion on concurrency issues; on transactional behavior, which explains why oak-online-alter-table performs correctly. Some of these are very relvant, I believe, to Mark’s suggestion. In particular, making the chunks copy; retaining transactional integrity, etc.

To remove any doubt, oak-online-alter-table is not production ready or anywhere near. Use at your own risk. I’ve seen it work, and I’ve seen it crash. I got little feedback and thus little chance to fix things. I also didn’t touch the code for quite a few months now, so I’m a little rusty myself.

.DEB or python packages are available, as usual, on project page on Google Code.

Thank you, Lenz!

• oak-online-alter-table: now works with multicolumn UNIQUE KEYs
• New: oak-chunk-update: break an long running or non indexed UPDATE/DELETE to small chunks
• oak-purge-master-logs, oak-show-replication-status: support for non standard slave ports

openark kit is a set of utilities for MySQL, helping in easing out everyday’s work. Let’s look more closely at the changes.

oak-online-alter-table

The utility allows for non-blocking ALTER TABLE operations, under certain limitations. One limitation which has been removed in the current release was the single-column UNIQUE KEY limitation. As of now, a requirement for running oak-online-alter-table is that the altered table has some UNIQUE KEY. It could be numerical, textual, single column, multi-column (compound), anything.

Other requirements:

• The table must not have any FOREIGN KEY association (neither parent nor child). This will be the next requirement to change. Support for child-side FOREIGN KEY will be added. At current, I do not see a solution for parent-side.
• The altered table must share a UNIQUE KEY with the original one (that is, at least one UNIQUE KEY must stay in place after altering the table).
• The table must not have any ‘AFTER’ triggers defined. At current, I do not see a solution for that, the reason being that the utility needs to create those triggers as port of its mechanism, and that MySQL does not allow for atomic alteration of triggers. Also see this post.
• Some operations, such as TRUNCATE, OPTIMIZE etc. are not permitted on the table while the utility works.

There was need (and there still is) for many MySQL issues workarounds. I’ll update as work progresses.

oak-chunk-update

Using similar techniques to oak-online-alter-table, and requiring a UNIQUE KEY on a table, this utility allows for breaking of a long running, or a non-indexed query, into small chunks, thereby allowing for relatively non blocking UPDATE/DELETE operations.

For example, a DELETE FROM log WHERE log_ts < ’2009-01-01 00:00:00′ might prove to be a long running query. If the log_ts column is not indexed, the table is virtually locked, whether this was MyISAM or InnoDB. Assuming an AUTO_INCREMENT column called ‘id’ exists (any UNIQUE KEY will do, so this is just an example), we can provide the following query as argument to oak-chunk-update:

DELETE FROM log WHERE log_ts < '2009-01-01 00:00:00' AND OAK_CHUNK(log.id)

The utility will parse the magic token OAK_CHUNK(…), and will work the query over distinct ranges, e.g.:

DELETE FROM log WHERE log_ts < '2009-01-01 00:00:00' AND log.id > 11000 AND log.id <= 12000
DELETE FROM log WHERE log_ts < '2009-01-01 00:00:00' AND log.id > 12000 AND log.id <= 13000
etc.

While the above can also be easily solved using a simple stored procedure, the magic becomes more interesting when you have a compound UNIQUE KEY, which may have non integral columns.

The utility also supports operations on multiple tables. Thus, one can use DELETE FROM t1 USING t1,t2…, for example, as long as an OAK_CHUNK(…) is proivded on one of the tables. It is interesting to see how the choice of table affects the chunking process.

oak-purge-master-logs, oak-show-replication-status

The two now support non default ports for slaves. This is achieved by utilizing the report_port parameter on the slaves, reading the value on SHOW SLAVE HOSTS (see this post). Both utilities support skipping SHOW SLAVE HOSTS and just picking up slaves by looking at SHOW PROCESSLIST.

Bugs

I’ve had issues with Google’s Issues, which made me leave a couple of unanswered issues for a couple of months. Brrrr. (BTW both were only related to prerequisites). I’ll resolve my issues issues. I’m looking forward for input, even though my spare time is little these days.

Acknowledgments

Writing of the toolkit is mostly done at my spare time, though being utilized in my professional work. Recently, a customer in urgent need has agreed for paying out for required development time on these scripts, while recognizing the open source model and license under which it is being developed and released, and removing all claims for copyright. I very much appreciate the customer’s approach and wish to acknowledge his contribution.

• The table has at least one single-column UNIQUE KEY [*]
• Altered table shares a single-column UNIQUE KEY with the original table [*]
• No ‘AFTER’ triggers are defined on the table (the utility creates its own triggers for the duration of the operation)
• The table has no FOREIGN KEYs [*][#]
• Table name is no longer than 57 characters

[*]: Restriction is scheduled to be removed or partly removed.

[#]: ‘Child-side’ foreign keys may actually work, but have not been tested.

Follows is a mini FAQ which attempts to introduce the utility.

So what exactly does this utility provide?

• First and foremost, the ability to perform a non blocking ALTER TABLE. This has long been an issue with MySQL, and complex Master-Master, application aware solutions are currently required in order to perform an ALTER TABLE with minimal downtime. The utility offers a no-downtime solution, albeit there is performance penalty for the duration of its runtime, and some requirements to meet.
• It also supports a ‘null’ ALTER. That is, an ALTER TABLE which does not change anything. This effectively means rebuilding of the table. For InnoDB tables with innodb_file_per_table, for example, this could be the means of regaining disk space after removing many rows from the table. Also, while it does not strictly act like OPTIMIZE TABLE, the effect of running this utility should build a better organized table on disk (this as yet unverified).
• Another thing this utility supports is the building of a ghost table: a duplicate of a given table, which keeps mirroring the original table via triggers. [May be removed in future versions]

How does it work?

Well, there are some wiki pages [1] [2] [3] which explain the details, but in general it goes as follows:

1. The utility creates an empty ‘ghost’  table, a copy of the original
2. The ghost table is ALTERed (or left unchanged, if no ALTER specified)
3. Triggers are created on the original table, which propagate changes to the ghost table.
4. In two passes, data is copied from the original table to the ghost table, then reviewed for deletion.
5. The ghost table is renamed to take place of the original table; the original table is dropped.

Here’s an invocation sample, in which we modify the type of the AUTO_INCREMENT column, add a column, add a key and drop a column – all at the same time, all online:

oak-online-alter-table -u root --ask-pass --socket=/tmp/mysql.sock --database=world --table=City --alter="MODIFY Id BIGINT UNSIGNED AUTO_INCREMENT, ADD COLUMN Mayor VARCHAR(64) CHARSET utf8, DROP COLUMN District, ADD KEY(Population)"

What kind of ALTERs are supported?

• ADD COLUMN (new column must have a default value)
• DROP COLUMN (as long as there’s still a shared single-column UNIQUE KEY between the old table and the altered one)
• MODIFY COLUMN (change type, including UNIQUE KEY columns)
• ADD KEY (normal, unique, fulltext etc.)
• DROP KEY (as long as there’s still a shared single-column UNIQUE KEY between the old table and the altered one)
• Change ENGINE: works, but great care should be taken when dealing with non-transactional engines
• Adding FOREIGN KEY constraints is possible
• More… Not all ALTERS have been tested, e.g. CONVERT, partition handling…
• none: it is possible to just do an online rebuild of the table, with no modification to its structure

The following are not allowed while the utility is running:

• An ALTER TABLE on the original table (well, obviously)
• A TRUNCATE on the original table
• LOAD DATA INFILE into the original table
• OPTIMIZE on the original table

In short, while this utility runs, just act normal…

Which engines are supported?

MyISAM & InnoDB are both supported. Any engine which uses table level locks is implicitly supported (Memory, Archive). There is currently no support for Maria (transactional), PBXT or Falcon. No plans for the near future include these.

It is possible to ALTER the ENGINE of a table, but this is highly unrecommended (due to differences in transactional behavior, or lack of transactional support altogether).

With InnoDB, and when running long transactions during the runtime of this utility, one of the steps (which requires a LOCK TABLES…WRITE) may take some time to complete: it repeatedly attempts to set up locks until successful. However, there is no blocking during these attempts, and the lock time, once the locks are gained, is minor.

What impact does this utility have?

For the duration of runtime, you can expect a performance penalty. There are many reasons for that:

• Triggers are added to your table. This, by itself, is performance penalty.
• Any changes to your table is propagated to the ghost table. So every INSERT makes for two, actually.
• More locks are held. While this utility is non-blocking, it does lock small chunks in turns, and for very short periods. But still locks are held.
• While the ghost table is being built, disk space of up to the same size as your original table is consumed. Depending on your ENGINE and storage means, some disk space may be released later on.

I was drunk when I decided to run it, but now that I’m sober, I find out it’s still running and I want to undo everything!

OK. First, don’t drink and DBA. Next, stop the utility, run it again with –cleanup. All should be back to normal.

How mature is this utility? I think there’s a bug!

It is very experimental. It seems to be working well as far as I know, but community feedback is appreciated. Please report bugs here. Thank you!

There is no such privilege as ‘LOGIN’ in the grants table, as the ability to log in is the most basic one MySQL allows. This basic privilege is called USAGE.

I’ll present a hack around this, one which oak-block-account implements. Before presenting the hack, lets lay down some requirements:

• A user can be blocked from logging in to MySQL.
• Such a blocked user can later be ‘released’, re-enabling him to log in.
• It should be possible to determine if a certain user is currently blocked or not.

A first attempt to answer the above requirements is to change the account’s password. As a naive example, we can set an account’s password to ‘aaaaaaaaa’. But let’s consider the following:

• Will the user be unable to find, by some algorithm or by brute force, the new password?
• How can we revert the new password to the original one?

Time to look at how MySQL stores passwords, then.

We begin by distinguishing old_passwords (variable old_passwords=1) from new passwords.

• Old passwords are 16 characters long. These are hexadecimal characters.
• New passwords are 41 characters long: a leading ‘*’ followed by 40 hexadecimal characters.

Blocking with ‘new passwords’

To disable an account using a ‘new password’, the trick is simply to reverse the password in the mysql.user table. So if my password was ’123456′ (strong one, isn’t it?), then mysql.user will have:

mysql> select PASSWORD('123456');
+-------------------------------------------+
| PASSWORD('123456')                        |
+-------------------------------------------+
| *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 |
+-------------------------------------------+

To encode, we do:

SET PASSWORD FOR 'some_user'@'some_host' = '9DA2AC2DE76CD7ADD8654EE50192347BE7384BB6*'

Let’s consider the implication of what we just did:

• The new password is valid, as far as MySQL is concerned. No questions asked.
• The user cannot log in with his old password.
• Nor can the user log in with any other password, since the PASSWORD() function will never return a password ending with ‘*’.
• It is easy to see that the user is ‘blocked’: his password ends with ‘*’.
• It is easy to restore the original password: we simply reverse the text and call SET PASSWORD again.

Blocking with ‘old passwords’

This part really assumes you’re using MySQL 4.1 or above. If you’re one of those ‘few’ lucky people, but are unfortunately using old_passwords, here’s the deal:

Reversing an old password won’t do, since:

• The reverse may still consist of an encoding for some password
• It’s impossible to tell if a user is blocked or not.

MySQL will only allow 16 or 41 character long passwords (anyway that’s my finding). So to encode a 16 characters long password, we pad it with 25 (= 41-16) ‘~’ characters. Thus, the encoded password ‘abcdef0123456789′ turns to ‘~~~~~~~~~~~~~~~~~~~~~~~~~abcdef0123456789′. Again, note the following:

• The new password is accepted by MySQL as valid.
• The user cannot log in with his old password.
• Nor can the user log in with any other password, since the PASSWORD() function will never return a password starting with ‘~’.
• It is easy to see that the user is ‘blocked’: his password starts with 25 ‘~’ characters.
• It is easy to restore the original password: we simply remove the leading ‘~’ characters and call SET PASSWORD again.

oak-block-account

No need to work all this by hand. oak-block-account is a utility which does exactly that. It can block, release, and even kill accounts. It will automatically detect if the password is ‘old’ or ‘new’, or if the account is already blocked or not.

Other possibilities

RENAME USER is another trick which could be used: we could take a user’s login (e.g. ‘webuser’) and rename it to an unknown value, like ‘webuser__1q98d4f’. While it serves the same purpose – of blocking the user, it has a disadvantage: if by luck or hack the new login is discovered, it could still be used to access the database. The change of password solution ensures there is no user/password combination which will work on the blocked account.

Other options may involve removing the account from mysql.user, to put it elsewhere, from where to restore the row when the time comes. I prefer a solution which works on the mysql schema itself.

The available tools are:

• oak-apply-ri: apply referential integrity on two columns with parent-child relationship.
• oak-block-account: block or release MySQL users accounts, disabling them or enabling them to login.
• oak-kill-slow-queries: terminate long running queries.
• oak-modify-charset: change the character set (and collation) of a textual column.
• oak-purge-master-logs: purge master logs, depending on the state of replicating slaves.
• oak-security-audit: audit accounts, passwords, privileges and other security settings.
• oak-show-limits: show AUTO_INCREMENT “free space”.
• oak-show-replication-status: show how far behind are replicating slaves on a given master.

All tools are written in Python, and require Python 2.3 or newer, and the python-mysqldb driver.

The project is hosted in Google Code, where you can find downloads, issue tracking etc. Community feedback is requested and welcome. Please use the ‘Issues’ mechanism to report bugs.

The openark kit is released under the BSD license.

All utilities have been put to work on production, yet all are still being developed, and still more are being created, as new ideas and needs emerge. I have been using custom made scripts for handling DB issues for years, and finally have decided to formalize them and to support them as an open source project. I do hope you try them out.