code.openark.org » Analysis

common_schema, rev. 178: foreach(), repeat_exec(), Roland Bouman, query analysis

shlomi — Thu, 01 Dec 2011 09:33:01 +0000

common_schema, revision 178 is now released, with major additions. This revision turns common_schema into a framework, rather than a set of views and functions.

common_schema provides with query scripting, analysis & informational views, and a function library, allowing for easier administration and diagnostics for MySQL. It introduces SQL based tools which simplify otherwise complex shell and client scripts, allowing the DBA to be independent of operating system, installed packages and dependencies.

There's no Perl nor Python, and no dependencies to install. It's just a schema.

Some highlights for the new revision:

foreach(), aka $(): loop through a collection, execute callback commands per element.
repeat_exec(): a repeat-until device: execute queries until some condition holds.
exec_file(): execute files a-la SOURCE, but on server side
Query analysis: analyze query text, view or routine definitions to detect dependency objects.
Improvements to views and routines, new routines introduced.

Let's take a closer look:

rpbouman

I'm very happy to have Roland Bouman working on this project. He introduced some sophisticated code without which some functionality could not take place. I'm sure I don't need to introduce his great capabilities; I'll just pass the note that it is very good working with him!

foreach()

Introducing a looping device which can iterate a collection and execute callback commands.

What's a collection? A range of numbers; a set of constants; the result set of a SELECT query; tables in your database and more.

What is a callback? A query or set of queries to invoke on the specific elements in the collection. For example:

call foreach('table in sakila', 'ALTER TABLE ${schema}.${table} ENGINE=InnoDB ROW_FORMAT=COMPRESSED');

I'll publish dedicated posts on foreach(), aka $(), following this post. Official documentation is here.

repeat_exec()

Repeat executing queries in a given interval, until some condition holds.

What kind of condition? You can loop forever, or until a given time has passed, a given number of iteration has passed.

You can iterate until no rows are affected by your commands (your callbacks), or until some dynamic condition holds (a query evaluates to true).

For example: purge rows from a table until no more rows are affected; in interval of 3 second:

call repeat_exec(3, 'DELETE FROM test.event WHERE ts < CURDATE() ORDER BY id LIMIT 1000', 0);

Official documentation is here.

exec_file()

If you need to execute commands from a file, you usually invoke SOURCE:

mysql> SOURCE '/tmp/somefile.sql';

Or you invoke mysql client and redirect its input to read from file:

bash$ mysql some_db < /tmp/somefile.sql

SOURCE is a MySQL client command. The file must reside on your client. Running the mysql client is great, but you need to work it out from outside the server.

call_exec() will let you import a file on server side, from within the server:

call exec_file('/tmp/some_file.sql');

You will need to have the file readable; it is limited to 64K at this moment; it may not use DELIMITER, and it may not include dynamic SQL. These are the limitations.

Official documentation is here.

exec() / exec_single()

All of the above rely on the exec() / exec_single() routines, which dynamically execute a set of queries. One one hand, it's no big deal: they only have to use prepared statements in order to invoke the queries. But then, they knows how to parse multiple queries (find the ";" delimiter correctly), plus they allow for configuration: if you set @common_schema_dryrun, queries are not actually executes; just printed out. If you set @common_schema_verbose, queries are verbosed in addition to being executed. Since all execution routines rely on these,we get a standardized pattern.

Official documentation is here.

Query analysis

Query parsing routines allow for detection of dependencies within queries. While not full-blown SQL parser, these allow one to realize on which tables or routines a view depends on; or a routines depends on; or an event; or just any query.

These routines can analyze the text of not only a SELECT query, but also UPDATE, DELETE, CREATE, etc. They can read the code of a stored routines, including queries and control flow constructs; thus, they are also able to analyze events and triggers.

At this stage forward-dependencies resolution is supported. This can eventually lead to dependency graphs or to reverse-dependency resolution (i.e. "which view, routine, trigger or event depend on table t?")

Example:

mysql> call get_view_dependencies('sakila', 'actor_info');
+-------------+---------------+-------------+--------+
| schema_name | object_name   | object_type | action |
+-------------+---------------+-------------+--------+
| sakila      | actor         | table       | select |
| sakila      | category      | table       | select |
| sakila      | film          | table       | select |
| sakila      | film_actor    | table       | select |
| sakila      | film_category | table       | select |
+-------------+---------------+-------------+--------+

The query analysis routines are in BETA stage.

Official documentation is here.

Test quite

common_schema is now tested. Not all code is as yet under tests; all new code is, and some of the older code. Work is in progress to add more and more tests.

Further changes:

candidate_keys does not give higher score for PRIMARY KEYs any longer. It ranks all unique keys according to its own heuristic; it also provides with the is_primary and is_nullable columns.
Added candidate_keys_recommended view, recommending best candidate key per table (while noting whether it qualifies as PRIMARY KEY in terms of NULLable columns).
Added many text parsing and text manipulation routines, such as better trim, tokenizing, etc. Improved existing code significantly.

Get it

common_schema is available for downloaded. It is released under the BSD license, and is free.

I've put very hard work into common_schema's documentation. It is very thorough and provides with clear examples. The documentation is also available for download.

If you encounter problems, please report on the issues page.

common_schema is meant to be downloaded & installed on any MySQL server. It provides with general and essential functionality. Spread the word!

MySQL Global status difference using single query

shlomi — Fri, 12 Aug 2011 16:31:12 +0000

Have just read MySQL Global status difference using MySQL procedures / functions, by Andres Karlsson. Have commented, but realized I did not provide with a direct answer. In the comment, I suggested checking out a solution based on views, found in common_schema. But the solution in common_schema is split into two views, due to the fact views cannot handle derived tables subqueries.

Well, here's a single query to do that: it checks GLOBAL_STATUS twice, 10 seconds apart in the following sample. It uses SLEEP() to actually wait between the two reads. Yes, you can do that with a query.

The following query shows all GLOBAL_STATUS values that have changed during the sample period.

SELECT STRAIGHT_JOIN
   LOWER(gs0.VARIABLE_NAME) AS variable_name,
   gs0.VARIABLE_VALUE AS variable_value_0,
   gs1.VARIABLE_VALUE AS variable_value_1,
   (gs1.VARIABLE_VALUE - gs0.VARIABLE_VALUE) AS variable_value_diff,
   (gs1.VARIABLE_VALUE - gs0.VARIABLE_VALUE) / 10 AS variable_value_psec,
   (gs1.VARIABLE_VALUE - gs0.VARIABLE_VALUE) * 60 / 10 AS
variable_value_pminute
FROM
   (
     SELECT
       VARIABLE_NAME,
       VARIABLE_VALUE
     FROM
       INFORMATION_SCHEMA.GLOBAL_STATUS
     UNION ALL
     SELECT
       '',
       SLEEP(10)
     FROM DUAL
   ) AS gs0
   JOIN INFORMATION_SCHEMA.GLOBAL_STATUS gs1 USING (VARIABLE_NAME)
WHERE
   gs1.VARIABLE_VALUE != gs0.VARIABLE_VALUE
;
+-----------------------------------+------------------+------------------+---------------------+---------------------+------------------------+
| variable_name                     | variable_value_0 | variable_value_1 | variable_value_diff | variable_value_psec | variable_value_pminute |
+-----------------------------------+------------------+------------------+---------------------+---------------------+------------------------+
| aborted_clients                   | 2210669          | 2210686          |                  17 |                 1.7 |                    102 |
| bytes_received                    | 53259933210      | 53260211104      |              277894 |             27789.4 |                1667364 |
| bytes_sent                        | 351130988015     | 351132884956     |             1896941 |            189694.1 |               11381646 |
| com_change_db                     | 3760546          | 3760584          |                  38 |                 3.8 |                    228 |
| com_delete                        | 6774784          | 6774801          |                  17 |                 1.7 |                    102 |
| com_insert                        | 52743750         | 52744012         |                 262 |                26.2 |                   1572 |
| com_insert_select                 | 13362650         | 13362740         |                  90 |                   9 |                    540 |
| com_select                        | 51722818         | 51723107         |                 289 |                28.9 |                   1734 |
| com_set_option                    | 108564134        | 108564754        |                 620 |                  62 |                   3720 |
| com_show_collations               | 3760530          | 3760568          |                  38 |                 3.8 |                    228 |
| com_show_processlist              | 366078           | 366082           |                   4 |                 0.4 |                     24 |
| com_show_status                   | 366047           | 366051           |                   4 |                 0.4 |                     24 |
| com_show_variables                | 3760535          | 3760573          |                  38 |                 3.8 |                    228 |
| com_update                        | 6271283          | 6271324          |                  41 |                 4.1 |                    246 |
| connections                       | 3781382          | 3781420          |                  38 |                 3.8 |                    228 |
| created_tmp_disk_tables           | 983223           | 983224           |                   1 |                 0.1 |                      6 |
| created_tmp_tables                | 9134044          | 9134126          |                  82 |                 8.2 |                    492 |
| handler_commit                    | 125798040        | 125798688        |                 648 |                64.8 |                   3888 |
| handler_delete                    | 6849562          | 6849578          |                  16 |                 1.6 |                     96 |
| handler_read_first                | 5332451          | 5332498          |                  47 |                 4.7 |                    282 |
| handler_read_key                  | 373910509        | 373912469        |                1960 |                 196 |                  11760 |
| handler_read_next                 | 850122025        | 850170403        |               48378 |              4837.8 |                 290268 |
| handler_read_rnd                  | 255104660        | 255105932        |                1272 |               127.2 |                   7632 |
| handler_read_rnd_next             | 992505444        | 992549948        |               44504 |              4450.4 |                 267024 |
| handler_update                    | 27930283         | 27930465         |                 182 |                18.2 |                   1092 |
| handler_write                     | 2051582925       | 2051602416       |               19491 |              1949.1 |                 116946 |
| innodb_buffer_pool_pages_data     | 77232            | 77243            |                  11 |                 1.1 |                     66 |
| innodb_buffer_pool_pages_dirty    | 626              | 645              |                  19 |                 1.9 |                    114 |
| innodb_buffer_pool_pages_flushed  | 38429812         | 38430032         |                 220 |                  22 |                   1320 |
| innodb_buffer_pool_pages_misc     | 4294922063       | 4294922052       |                 -11 |                -1.1 |                    -66 |
| innodb_buffer_pool_read_requests  | 1933796064       | 1933871603       |               75539 |              7553.9 |                 453234 |
| innodb_buffer_pool_reads          | 11360212         | 11360214         |                   2 |                 0.2 |                     12 |
| innodb_buffer_pool_write_requests | 1074109722       | 1074115394       |                5672 |               567.2 |                  34032 |
| innodb_data_fsyncs                | 5583880          | 5583905          |                  25 |                 2.5 |                    150 |
| innodb_data_read                  | 3339489280       | 3339501568       |               12288 |              1228.8 |                  73728 |
| innodb_data_reads                 | 11796492         | 11796494         |                   2 |                 0.2 |                     12 |
| innodb_data_writes                | 105587582        | 105588145        |                 563 |                56.3 |                   3378 |
| innodb_data_written               | 3721600000       | 3727315968       |             5715968 |            571596.8 |               34295808 |
| innodb_dblwr_pages_written        | 38429812         | 38430032         |                 220 |                  22 |                   1320 |
| innodb_dblwr_writes               | 596503           | 596506           |                   3 |                 0.3 |                     18 |
| innodb_log_write_requests         | 380978894        | 380981368        |                2474 |               247.4 |                  14844 |
| innodb_log_writes                 | 74407604         | 74407990         |                 386 |                38.6 |                   2316 |
| innodb_os_log_fsyncs              | 2310799          | 2310807          |                   8 |                 0.8 |                     48 |
| innodb_os_log_written             | 2905292800       | 2906502656       |             1209856 |            120985.6 |                7259136 |
| innodb_pages_created              | 1341584          | 1341593          |                   9 |                 0.9 |                     54 |
| innodb_pages_read                 | 13117652         | 13117654         |                   2 |                 0.2 |                     12 |
| innodb_pages_written              | 38429812         | 38430032         |                 220 |                  22 |                   1320 |
| innodb_rows_deleted               | 6849552          | 6849568          |                  16 |                 1.6 |                     96 |
| innodb_rows_inserted              | 43787980         | 43788207         |                 227 |                22.7 |                   1362 |
| innodb_rows_read                  | 4289845136       | 4289919560       |               74424 |              7442.4 |                 446544 |
| innodb_rows_updated               | 24119627         | 24119809         |                 182 |                18.2 |                   1092 |
| key_read_requests                 | 41262330         | 41262338         |                   8 |                 0.8 |                     48 |
| open_files                        | 7                | 5                |                  -2 |                -0.2 |                    -12 |
| opened_files                      | 4212920          | 4212924          |                   4 |                 0.4 |                     24 |
| questions                         | 253158874        | 253160331        |                1457 |               145.7 |                   8742 |
| select_full_join                  | 546              | 547              |                   1 |                 0.1 |                      6 |
| select_range                      | 721945           | 721947           |                   2 |                 0.2 |                     12 |
| select_scan                       | 12828865         | 12828989         |                 124 |                12.4 |                    744 |
| sort_range                        | 170971           | 170973           |                   2 |                 0.2 |                     12 |
| sort_rows                         | 255175383        | 255176655        |                1272 |               127.2 |                   7632 |
| sort_scan                         | 534078           | 534080           |                   2 |                 0.2 |                     12 |
| table_locks_immediate             | 142673687        | 142674454        |                 767 |                76.7 |                   4602 |
| threads_cached                    | 7                | 8                |                   1 |                 0.1 |                      6 |
| threads_connected                 | 5                | 10               |                   5 |                 0.5 |                     30 |
| threads_created                   | 840486           | 840509           |                  23 |                 2.3 |                    138 |
+-----------------------------------+------------------+------------------+---------------------+---------------------+------------------------+

Some values don't make sense to do difference on (e.g. threads_connected), since they present with momentary status and are not incrementing as others (e.g. threads_created).

Happy SQLing!

Announcing common_schema: common views & routines for MySQL

shlomi — Wed, 13 Jul 2011 04:25:24 +0000

Today I have released common_schema, a utility schema for MySQL which includes many views and functions, and is aimed to be installed on any MySQL server.

What does it do?

There are views answering for all sorts of useful information: stuff related to schema analysis, data dimensions, monitoring, processes & transactions, security, internals... There are basic functions answering for common needs.

Some of the views/routines simply formalize those queries we tend to write over and over again. Others take the place of external tools, answering complex questions via SQL and metadata. Still others help out with SQL generation.

Here are a few highlights:

Did you know you can work out simple monitoring of your server with a query? There's a view to do that for you.
How about showing just the good parts of the processlist?
Does your schema have redundant keys?
Or InnoDB tables with no PRIMARY KEY?
Is AUTO_INCREMENT running out of space?
Can I get the SQL statements to generate my FOREIGN KEYs? To drop them?
And can we finally get SHOW GRANTS for all accounts, and as an SQL query?
Ever needed a 64 bit CRC function?
And aren't you tired of writing the cumbersome SUBSTRING_INDEX(SUBSTRING_INDEX(str, ',', 3), ',', -1)? There's an alternative.

There's more. Take a look at the common_schema documentation for full listing. And it's evolving: I've got quite a few ideas already for future components.

Some of these views rely on heavyweight INFORMATION_SCHEMA tables. You should be aware of the impact and risks.

What do I need to install?

There's no script or executable file. It's just a schema. The distribution in an SQL file which generates common_schema. Much like a dump file.

What are the system requirements?

It's just between you and your MySQL. There are currently three distribution files, dedicated for different versions of MySQL (and allowing for increased functionality):

common_schema_mysql_51: fits all MySQL >= 5.1 distributions
common_schema_innodb_plugin: fits MySQL >= 5.1, with InnoDB plugin + INFORMATION_SCHEMA tables enabled
common_schema_percona_server: fits Percona Server >= 5.1

Refer to the documentation for more details.

What are the terms of use?

common_schema is released under the BSD license.

Where can I download it?

On the common_schema project page. Enjoy it!

Checking for AUTO_INCREMENT capacity with single query

shlomi — Tue, 05 Apr 2011 05:36:56 +0000

Darn! This means oak-show-limits becomes redundant. Am I not supposed to speak about it on my coming presentation? Bad timing!

You have AUTO_INCREMENT columns. How far are you pushing the limits? Are you going to run out of AUTO_INCREMENT values soon? Perhaps you wonder whether you should ALTER from INT to BIGINT?

The answer is all there in INFORMATION_SCHEMA. The TABLES table shows the current AUTO_INCREMENT value per table, and the COLUMNS table tells us all about a column's data type.

It takes some ugly code to deduce the maximum value per column type, what with signed/unsigned and data type, but then its very simple. Here is the query:

SELECT
  TABLE_SCHEMA,
  TABLE_NAME,
  COLUMN_NAME,
  DATA_TYPE,
  COLUMN_TYPE,
  IF(
    LOCATE('unsigned', COLUMN_TYPE) > 0,
    1,
    0
  ) AS IS_UNSIGNED,
  (
    CASE DATA_TYPE
      WHEN 'tinyint' THEN 255
      WHEN 'smallint' THEN 65535
      WHEN 'mediumint' THEN 16777215
      WHEN 'int' THEN 4294967295
      WHEN 'bigint' THEN 18446744073709551615
    END >> IF(LOCATE('unsigned', COLUMN_TYPE) > 0, 0, 1)
  ) AS MAX_VALUE,
  AUTO_INCREMENT,
  AUTO_INCREMENT / (
    CASE DATA_TYPE
      WHEN 'tinyint' THEN 255
      WHEN 'smallint' THEN 65535
      WHEN 'mediumint' THEN 16777215
      WHEN 'int' THEN 4294967295
      WHEN 'bigint' THEN 18446744073709551615
    END >> IF(LOCATE('unsigned', COLUMN_TYPE) > 0, 0, 1)
  ) AS AUTO_INCREMENT_RATIO
FROM
  INFORMATION_SCHEMA.COLUMNS
  INNER JOIN INFORMATION_SCHEMA.TABLES USING (TABLE_SCHEMA, TABLE_NAME)
WHERE
  TABLE_SCHEMA NOT IN ('mysql', 'INFORMATION_SCHEMA', 'performance_schema')
  AND EXTRA='auto_increment'
;

There's one row in the result set for each AUTO_INCREMENT column. since at most one AUTO_INCREMENT column can exist for any given table, each row also identifies a unique table. Resulting columns are mostly self-explanatory, but here's some details on some of the columns:

IS_UNSIGNED: 1 when the column is UNSIGNED, 0 otherwise.
MAX_VALUE: maximum value that can be contained within column.
AUTO_INCREMENT: current AUTO_INCREMENT value for table.
AUTO_INCREMENT_RATIO: value in the range [0..1], where 1 means "100% full".

A sample output:

+--------------+------------+--------------+-----------+-----------------------+-------------+------------+----------------+----------------------+
| TABLE_SCHEMA | TABLE_NAME | COLUMN_NAME  | DATA_TYPE | COLUMN_TYPE           | IS_UNSIGNED | MAX_VALUE  | AUTO_INCREMENT | AUTO_INCREMENT_RATIO |
+--------------+------------+--------------+-----------+-----------------------+-------------+------------+----------------+----------------------+
| sakila       | actor      | actor_id     | smallint  | smallint(5) unsigned  |           1 |      65535 |            201 |               0.0031 |
| sakila       | address    | address_id   | smallint  | smallint(5) unsigned  |           1 |      65535 |            606 |               0.0092 |
| sakila       | category   | category_id  | tinyint   | tinyint(3) unsigned   |           1 |        255 |             17 |               0.0667 |
| sakila       | city       | city_id      | smallint  | smallint(5) unsigned  |           1 |      65535 |            601 |               0.0092 |
| sakila       | country    | country_id   | smallint  | smallint(5) unsigned  |           1 |      65535 |            110 |               0.0017 |
| sakila       | customer   | customer_id  | smallint  | smallint(5) unsigned  |           1 |      65535 |            600 |               0.0092 |
| sakila       | film       | film_id      | smallint  | smallint(5) unsigned  |           1 |      65535 |           1001 |               0.0153 |
| sakila       | inventory  | inventory_id | mediumint | mediumint(8) unsigned |           1 |   16777215 |           4582 |               0.0003 |
| sakila       | language   | language_id  | tinyint   | tinyint(3) unsigned   |           1 |        255 |              7 |               0.0275 |
| sakila       | payment    | payment_id   | smallint  | smallint(5) unsigned  |           1 |      65535 |          16050 |               0.2449 |
| sakila       | rental     | rental_id    | int       | int(11)               |           0 | 2147483647 |          16050 |               0.0000 |
| sakila       | staff      | staff_id     | tinyint   | tinyint(3) unsigned   |           1 |        255 |              3 |               0.0118 |
| sakila       | store      | store_id     | tinyint   | tinyint(3) unsigned   |           1 |        255 |              3 |               0.0118 |
+--------------+------------+--------------+-----------+-----------------------+-------------+------------+----------------+----------------------+

Bonus: free advice on increasing your AUTO_INCREMENT capacity

Make it UNSIGNED. No, really. Check your definitions now.

oak-hook-general-log: your poor man's Query Analyzer

shlomi — Wed, 15 Dec 2010 17:46:06 +0000

The latest release of openark kit introduces oak-hook-general-log, a handy tool which allows for some analysis of executing queries.

Initially I just intended for the tool to be able to dump the general log to standard output, from any machine capable to connect to MySQL. Quick enough, I realized the power it brings.

With this tool, one can dump to standard output all queries using temporary tables; or using a specific index; or doing a full index scan; or just follow up on connections; or... For example, the following execution will only log queries which make for filesort:

oak-hook-general-log --user=root --host=localhost --password=123456 --filter-explain-filesort

The problem with using the standard logs

So you have the general log, which you don't often enable, since it tends to grow huge within moments. You then have the slow log. Slow log is great, and is among the top tools for MySQL diagnosis.

The slow log allows for log-queries-not-using-indexes, which is yet another nice feature. Not only should you log any query running for over X seconds, but also log any query which does not use an index.

Wait. This logs all single-row tables (no single row table will use an index), as well as very small tables (a common 20 rows lookup table will most often be scanned). These are OK scans. This makes for some noise in the slow log.

And how about queries which do use an index, but do so poorly? They use an index, but retrieve some 12,500,000 rows, using temporary table & filesort?

What oak-hook-general-log does for you

This tool streams out the general log, and filters out queries based on their role or on their execution plan.

To work at all, it must enable the general log. Moreover, it directs the general log to log table. Mind that this makes for a performance impact, which is why the tool auto-terminates and restores original log settings (default is 1 minute, configurable). It's really not a tool you should keep running for days. But during the few moments it runs, it will:

Routinely rotate the mysql.general_log table so that it doesn't fill up
Examine entries found in the general log
Cross reference entries to the PROCESSLIST so as to deduce database context (bug #52554)
If required and appropriate, evaluate a query's execution plan
Decide whether to dump each entry based on filtering rules

Filtering rules

Filtering rules are passed as command line options. At current, only one filtering rule applies (if more than one specified only one is used, so no point in passing more than one). Some of the rules are:

filter-connection: only log connect/quit entries
filter-explain-fullscan: only log full table scans
filter-explain-temporary: only log queries which create implicit temporary tables
filter-explain-rows-exceed: only log queries where more than X number of rows are being accessed on some table (estimated)
filter-explain-total-rows-exceed: only log queries where more than X number of rows are accessed on all tables combined (estimated, with possibly incorrect numbers on some queries)
filter-explain-key: only log queries using a specific index. This feature somewhat overlaps with Maatkit's mk-index-usage (read announcement).
filter-explain-contains: a general purpose grep on the execution plan. Log queries where the execution plan contains some text.

There are other filters, and I will possibly add more in due time.

Here are a couple cases I used oak-hook-general-log for:

Use case: temporary tables

I have a server with this alarming chart (courtesy mycheckpoint) of temporary tables:

What could possibly create 30 temporary tables per second on average?

The slow log produced nothing helpful, even with log-queries-not-using-indexes enabled. There were a lot of queries not using indexes there, but nothing at these numbers. With:

oak-hook-general-log --filter-explain-temporary

enabled for 1 minute, nothing came out. Weird. Enabled for 5 minutes, I got one entry. Turned out a scheduled script, acting once per 5 minutes, was making a single complicated query involving many nested views, which accounted for some hundreds of temporary tables created. All of them very small, query time was very fast. There is no temporary tables problem with this server, case closed.

Use case: connections

A server had issues with some exceptions being thrown on the client side. There was a large number of new connections created per second although the client was using a connection pool. Suspecting the pool didn't work well, I issued:

oak-hook-general-log --filter-connect

The pool was working well, all right. No entries for that client were recorder in 1 minute of testing. However, it turned out some old script was flooding the MySQL server with requests, every second. The log showed root@somehost, and sure enough, the script was disabled. Exceptions were due to another reason; it was good to eliminate a suspect.

Some of the tool's use case is relatively easy to solve with tail, grep & awk; others are not. I am using it more and more often, and find it to make significant shortcuts in tracking down queries.

Get it

Download the tool as part of openark kit: access the openark kit project page.

Or get the source code directly.

Feedback is most welcome.

openark-kit (rev. 170): new tools, new functionality

shlomi — Wed, 15 Dec 2010 06:31:24 +0000

I'm pleased to announce a new release of the openark kit. There's a lot of new functionality inside; following is a brief overview.

The openark kit is a set of utilities for MySQL. They solve everyday maintenance tasks, which may be complicated or time consuming to work by hand.

It's been a while since the last announced release. Most of my attention was on mycheckpoint, building new features, writing documentation etc. However my own use of openark kit has only increased in the past few months, and there's new useful solutions to common problems that have been developed.

I've used and improved many tools over this time, but doing the final cut, along with proper documentation, took some time. Anyway, here are the highlights:

New tool: oak-hook-general-log

oak-hook-general-log hooks up a MySQL server and dumps the general log based on filtering rules, applying to query role or execution plan. It is possible to only dump connect/disconnect entries, queries which make a full table scan, or use temporary tables, or scan more than X number of rows, or...

I'll write more on this tool shortly.

New tool: oak-prepare-shutdown

This tool makes for an orderly and faster shutdown by safely stopping replication, and flushing InnoDB pages to disk prior to shutting down (keeping server available for connections even while attempting to flush dirty pages to disk). A typical use case would be:

oak-prepare-shutdown --user=root --ask-pass --socket=/tmp/mysql.sock && /etc/init.d/mysql stop

New tool: oak-repeat query

oak-repeat-query repeats executing a given query until some condition holds. The condition can be:

Number of given iterations has been reached
Given time has elapsed
No rows have been affected by query

The tool comes in handy for cleanup jobs, warming up caches, etc.

New tool: oak-get-slave-lag

This simple tool just returns the number of seconds a slave is behind master. But it also returns with an appropriate exit code, based on a given threshold: 0 when lag is good, 1 (error exit code) when lag is too great or slave fails to replicate.

This tool has been used by 3rd party applications, such as a load balancer, to determine whether a slave should be accessed.

Updated tool: oak-chunk-update

This extremely useful utility breaks down very long queries into smaller chunks. These could be queries which should affect a huge amount of rows, or queries which cannot utilize an index.

Updates to the tool include limiting the range of rows the tool scans, by specifying start and stop position (either by providing constant values or by SELECT query). Also added is auto-termination when no rows are found to be affected. Last, it is possible to override INFORMATION_SCHEMA lookup by explicitly specifying chunking key.

This tool works great for your daily/weekly/monthly batch jobs; in creating DWH tables; populating new columns; purging old entries; clearing data based on non-indexed values; generating summary tables; and more.

Frozen tool: oak-apply-ri

I haven't been using this tool for a while. The main work down by this tool can be done with oak-chunk-update. There are some additional safety checks oak-apply-ri provides; I'm thinking over if they justify the tool's existence.

Frozen tool: oak-online-alter-table

With the appearance of Facebook’s Online Schema Change (OSC) tool, which derives from oak-online-alter-table, I'm not sure I will continue developing the tool. I intend to wait for general feedback on OSC before making a decision.

Documentation

Documentation is now part of openark kit's SVN repository.

Download

The openark kit project is currently hosted by Google Code. Downloads are available at the Google Code openark kit project page.

Downloads are available in the following packaging formats:

.deb package, to be installed on debian, ubuntu and otherwise debian based distributions.
.rpm package, architecture free (noarch), for RPM supporting Linux distributions such as RedHat, Fedora, CentOS etc.
.tar.gz using python's distutils installer.
source, directly retrieved from SVN or from above python package.
Some distribution specific RPM packages, courtesy Lenz Grimmer.

Feedback

Your feedback is welcome! I may not always respond promptly; and I confess that some bugs were left open for more than I would have liked them to. I hope to make for good quality of code, and bug reporting is one major factor you can control.

EXPLAIN: missing db info

shlomi — Tue, 11 May 2010 04:57:02 +0000

I'm further developing a general log hook, which can stream queries from the general log.

A particular direction I'm taking is to filter queries by their type of actions. For example, the tool (oak-hook-general-log) can be instructed to only stream out those queries which involve creation of a temporary table; or those which cause for a filesort, or full index scan, etc.

This is done by evaluating of query execution plans on the fly. I suspect the MySQL query analyzer roughly does the same (as a small part of what it does).

It's almost nothing one cannot do with sed/awk. However, I bumped into a couple of problems:

The general log (and the mysql.general_log table, in particular) does not indicate the particular database one is using for the query. Since slow log does indicate this data, I filed a bug on this. I currently solve this by crossing connection id with the process list, where the current database is listed. It's shaky, but mostly works.
Just realized: there's no DB info in the EXPLAIN output! It's weird, since I've been EXPLAINing queries for years now. But I've always had the advantage of knowing the schema used: either because I was manually executing the query on a known schema, or mk-query-digest was kind enough to let me know.

For example, look at the following imaginary query, involving both the world and sakila databases:

mysql> use test;
Database changed
mysql> EXPLAIN SELECT * FROM world.Country JOIN sakila.city WHERE Country.Capital = city.city_id;
+----+-------------+---------+--------+---------------+---------+---------+-----------------------+------+-------------+
| id | select_type | table   | type   | possible_keys | key     | key_len | ref                   | rows | Extra       |
+----+-------------+---------+--------+---------------+---------+---------+-----------------------+------+-------------+
|  1 | SIMPLE      | Country | ALL    | NULL          | NULL    | NULL    | NULL                  |  239 |             |
|  1 | SIMPLE      | city    | eq_ref | PRIMARY       | PRIMARY | 2       | world.Country.Capital |    1 | Using where |
+----+-------------+---------+--------+---------------+---------+---------+-----------------------+------+-------------+
2 rows in set (0.00 sec)

The query is imaginary, since the tables don't actually have anything in common. But look at the EXPLAIN result: can you tell where city came from? Country can somehow be parsed from the ref column, but no help on city.

Moreover, table aliases show on the EXPLAIN plan (which is good), but with no reference to the original table.

So, is it back to parsing of the SQL query? I'm lazy reluctant to do that. It's error prone, and one needs to implement, or use, a good parser, which also accepts MySQL dialect. Haven't looked into this yet.

I'm currently at a standstill with regard to automated query execution plan evaluation where database cannot be determined.

Static charts vs. interactive charts

shlomi — Tue, 02 Mar 2010 13:28:08 +0000

I'm having my usual fun with charts. Working on mycheckpoint, I've generated monitoring charts using the Google Chars API. But I've also had chance to experiment and deploy interactive charts, JavaScript based. In particular, I used and tweaked dygraphs.

I'd like to note some differences in using charts of both kinds. And I think it makes a very big difference.

Static charts

I'll call any image-based chart by "static chart". It's just a static image. Example of such charts are those generated by Google Image Charts (they now also have new, interactive charts), or RRDtool. Show below is an example of a static chart; in this example, generated by Google:

Pros and cons of static charts

Pros

Images can be viewed on any graphical platform. Browsers, email clients, cell phones, whatever.
Self contained: chart image, legend, scales: all in one image.
As such, easy to move around.
Are safe to use.

Cons

Images are fuzzy. Is the com_replace_psec really 0? Maybe it's 0.1? A larger value can make lower values hard to tell.
Images are inaccurate: the colors can lie. The red and green lines showing are hard to tell apart. The red is painted above the green. Data gets "lost".
They do not zoom (one needs to regenerate larger image)
Unless encoded with base64, HTML pages which include images need to link outside.
In the particular case of Google Charts, one is limited to 2K length URL. Trust me, it's a big limitation! (PS, Google now support POST method to allow for up to 16K. But... it's a POST method...)
In the particular case of Google Charts, one must have an internet connection.
In the particular case of Google Charts, one must submit data to Google.

Interactive charts

Interactive charts are those which react to your commands. These are either JavaScript or Flash based, mostly. They allow for really nice features. Take the following chart as an example: try and move over with your mouse; or select sections to zoom in.

DML
[graphDiv]
[labelsDiv]

The above chart is generated with dygraphs. Since it is embedded within my WordPress page, the layout is affected by that of my theme. Take a look at this example page to see similar charts outside this blog site (Internet Explorer users: Maxmimize/minimize button will not work well for now. And, may I suggest Mozilla Firefox?)

Pros and cons of interactive charts

Pros

Can present you with exact values. No more doubt about the com_replace_psec values.
Can allow for zoom in, zoom out.

Cons

Need supporting platform. The above cannot be viewed by non-JavaScript browsers (cell phones, etc.)
Browser support is also an issue with JavaScript.
Emailing such report will result in mail blocking in many companies: mail filters will not allow for JavaScript code to pass.
Charts are not necessarily self-contained, in terms of the chart entity With Flash charts (e.g. Fusion Charts) this works. But in the above, the legend and scales are outside the image. As such, they cannot be just moved around.
HTML pages which include such charts can be self contained. The HTML page can include all the JavaScript dependencies, in addition to the chart generating code. Flash based charts cannot be self contained.

Summary

Interactive charts are cool!

I'm now integrating dygraphs into mycheckpoint (How nice it is to work with BSD & MIT licenses!). Though I may later switch to flot, interactive charts will be the next standard charting way in mycheckpoint. I will continue supporting static Google Charts, as follows from the above pros and cons list.

Performance analysis with mycheckpoint

shlomi — Thu, 12 Nov 2009 10:47:00 +0000

mycheckpoint (see announcement) allows for both graph presentation and quick SQL access to monitored & analyzed data. I'd like to show the power of combining them both.

InnoDB performance

Taking a look at one of the most important InnoDB metrics: the read hit ratio (we could get the same graph by looking at the HTML report):

SELECT innodb_read_hit_percent FROM sv_report_chart_sample \G
*************************** 1. row ***************************
innodb_read_hit_percent: http://chart.apis.google.com/chart?cht=lc&chs=400x200&chts=303030,12&chtt=Nov+10,+11:40++-++Nov+11,+08:55+(0+days,+21+hours)&chdl=innodb_read_hit_percent&chdlp=b&chco=ff8c00&chd=s:400664366P6674y7176677677u467773y64ux166666764366646y616666666666644444434444s6u4S331444404433341334433646777666666074736777r1777767764776666F667777617777777777777777yaRi776776mlf667676xgx776766rou67767777u37797777x76676776u6A737464y67467761777666643u66446&chxt=x,y&chxr=1,99.60,100.00&chxl=0:||Nov+10,+15:55|Nov+10,+20:10|Nov+11,+00:25|Nov+11,+04:40|&chxs=0,505050,10

We see that read hit is usually high, but occasionally drops low, down to 99.7, or even 99.6. But it seems like most of the time we are above 99.95% read hit ratio. It's hard to tell about 99.98%.

Can we know for sure?

We can stress our eyes, yet be certain of little. It's best if we just query for the metrics! mycheckpoint provides with all data, accessible by simple SQL queries:

SELECT SUM(innodb_read_hit_percent > 99.95)/count(*)
  FROM sv_report_sample;
+-----------------------------------------------+
| SUM(innodb_read_hit_percent > 99.95)/count(*) |
+-----------------------------------------------+
|                                        0.7844 |
+-----------------------------------------------+

Yes, most of the time we're above 99.95% read hit ratio: but not too often!

I'm more interested in seeing how much time my server's above 99.98% read hit:

SELECT SUM(innodb_read_hit_percent > 99.98)/count(*)
  FROM sv_report_sample;
+-----------------------------------------------+
| SUM(innodb_read_hit_percent > 99.98)/count(*) |
+-----------------------------------------------+
|                                        0.3554 |
+-----------------------------------------------+

We can see the server only has 99.98% read hit percent 35% of the time. Need to work on that!

Disk activity

Lower read hit percent means higher number of disk reads; that much is obvious. The first two following graphs present this obvious connection. But the third graph tells us another fact: with increased disk I/O, we can expect more (and longer) locks.

Again, this should be very intuitive, when thinking about it this way. The problem sometimes arises when we try to analyze it the other way round: "Hey! InnoDB has a lot of locks! What are we going to do about it?". Many times, people will look for answers in their transactions, their Isolation Level, their LOCK IN SHARE MODE clauses. But the simple answer can be: "There's a lot of I/O, so everything has to wait; therefore we increase the probability for locks; therefore there's more locks".

The answer, then, is to reduce I/O. The usual stuff: slow queries; indexing; ... and, yes, perhaps transactions or tuning.

The charts below make it quite clear that we have an issue of excessive reads -> less read hit -> increased I/O -> more locks.

By the way, the above resulted from the fact that, due to a problematic query, all slave stopped replicating. Slaves participated in read-balancing, so when they went stale, all reads were directed at the master (the monitored node).

You have the metrics at your disposal

Looking at the following chart:

It appears that there's no slow queries. But this may be misleading: perhaps there's just a little, that don't show due to the chart's large scale?

One could argue that this is the chart's fault. Perhaps there should be a distinct chart for "slow queries percent". Perhaps I'll add one. But we can't have special charts for everything. It's would be too tiresome to look at hundreds of charts.

Anyway, my point is: let's verify just how many slow queries we have:

SELECT slow_queries_psec FROM sv_hour ORDER BY id DESC;
+-------------------+
| slow_queries_psec |
+-------------------+
|              3.05 |
|              3.83 |
|              4.39 |
|              4.03 |
|              3.86 |
|              3.56 |
|              3.73 |
|              3.79 |
|              3.58 |
|              3.55 |
...
+-------------------+

So, between 3 and 4 slow queries per second. It doesn't look too good in this light. Checking on the percentage of slow queries (of total questions):

SELECT ROUND(100*slow_queries_diff/questions_diff, 1) AS slow_queries_percent
  FROM sv_hour ORDER BY id DESC LIMIT 10;

Or, since the above calculation is pre-defined in the reports tables:

SELECT slow_queries_percent FROM sv_report_hour_recent;
+----------------------+
| slow_queries_percent |
+----------------------+
|                  0.8 |
|                  1.0 |
|                  1.2 |
|                  1.2 |
|                  1.1 |
|                  1.0 |
|                  1.1 |
|                  1.1 |
|                  1.0 |
...
+----------------------+

Accessible data

This is what I've been trying to achieve with mycheckpoint. As a DBA, consultant and SQL geek I find that direct SQL access works best for me. It's like loving command line interface over GUI tools. Direct SQL gives you so much more control and information.

Charting is important, since it's easy to watch and get first impressions, or find extreme changes. But beware of relying on charts all the time. Scale issues, misleading human interpretation, technology limitations - all these make charts inaccurate.

mycheckpoint allows for both methods, and, I believe, intuitively so.

Replication analysis with mycheckpoint

shlomi — Wed, 11 Nov 2009 03:50:28 +0000

I would like to show how mycehckpoint (see announcement) can be put to use for analyzing various replication metrics.

Lagging slaves

A slave has been monitored. Monitoring started at a time when it was way behind master (about two days lag), but it has since caught up. This can be easily verified by the following chart:

The above chart can be obtained by viewing the HTML report:

SELECT html FROM sv_report_html

Or by directly issuing the query:

mysql> SELECT seconds_behind_master FROM sv_report_chart_hour\G
*************************** 1. row ***************************
seconds_behind_master: http://chart.apis.google.com/chart?cht=lc&chs=400x200&chts=303030,12&chtt=Nov+5,+10:00++-++Nov+10,+08:00+(4+days,+22+hours)&chdl=seconds_behind_master&chdlp=b&chco=ff8c00&chd=s:976431zzzywutrpnliiifdbZYXVTRRRPNLJHEBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA&chxt=x,y&chxr=1,0,169811&chxl=0:||Nov+6,+09:00|Nov+7,+09:00|Nov+8,+08:00|Nov+9,+08:00|&chxs=0,505050,10

This is all nice. But I'm also interested in the rate at which slave lag decreased. Many ignore this important metric: just how fast does your slave replicate?

Slave catchup speed

So we're looking for the slope of the above graph. Luckily, the slope is nothing but the difference in seconds_behind_master divided by elapsed time. mycheckpoint supports the *_psec metrics for all known variables.

Again, we can just look at the HTML report, or we can issue:

mysql> SELECT seconds_behind_master_psec FROM sv_report_chart_hour\G
*************************** 1. row ***************************
seconds_behind_master_psec: http://chart.apis.google.com/chart?cht=lc&chs=400x200&chts=303030,12&chtt=Nov+5,+10:00++-++Nov+10,+08:00+(4+days,+22+hours)&chdl=seconds_behind_master_psec&chdlp=b&chco=ff8c00&chd=s:fbdebXU0wvOgaYXVUIutFTacefXXS4xhYRTJAKf90444444444444444444444482444444445444444444444472544444444444444444444481544444&chxt=x,y&chxr=1,-2.35,0.19&chxl=0:||Nov+6,+09:00|Nov+7,+09:00|Nov+8,+08:00|Nov+9,+08:00|&chxs=0,505050,10

And get this chart:

The numbers show a reasonable slave catchup speed (lower is better). For example, a -1 value indicates that the slave is replicating twice as fast as the master. So if we're 10 minutes behind, catch up will occur 10 minutes from now (the slave will cover for the missing 10 minutes plus the future 10 minutes by which the master will be writing). At times, replication went very fast (slope as low as -2.25 on average), and at times is was very slow (-0.25).

But all negative numbers are good! They mean slave is faster than master. Indeed, once the slave caught up, numbers climbed to 0: the slave has since been running at same speed as the master.

Estimating slave catchup time

When we have a lagging slave, there's nothing more interesting to known than "when will it finally catch up?". We can do the math: calculating slope, extrapolating the expected point in time where the lag drops to 0; or we can just ask!

Let's first ask the hard way, then the human way:

Querying the history of estimations:

SELECT ts, estimated_slave_catchup_seconds FROM sv_report_hour;
+---------------------+---------------------------------+
| ts                  | estimated_slave_catchup_seconds |
+---------------------+---------------------------------+
| 2009-11-06 16:00:00 |                          142984 |
| 2009-11-06 17:00:00 |                           47988 |
| 2009-11-06 18:00:00 |                           31416 |
| 2009-11-06 19:00:00 |                           22896 |
| 2009-11-06 20:00:00 |                           20202 |
| 2009-11-06 21:00:00 |                           13062 |
| 2009-11-06 22:00:00 |                            7932 |
| 2009-11-06 23:00:00 |                            5220 |
| 2009-11-07 00:00:00 |                            2740 |
| 2009-11-07 01:00:00 |                            NULL |
| 2009-11-07 02:00:00 |                             463 |
| 2009-11-07 03:00:00 |                            NULL |
| 2009-11-07 04:00:00 |                            NULL |
| 2009-11-07 05:00:00 |                            NULL |
| 2009-11-07 06:00:00 |                            NULL |
...
+---------------------+---------------------------------+

(NULL occurs when slave has caught up; so nothing to estimate)

Had we been checking up on human reports at the time of lag, we would see something like:

mysql> SELECT report FROM sv_report_human_hour ORDER BY ID DESC LIMIT 1\G
*************************** 1. row ***************************
report:
Report period: 2009-11-06 16:00:00 to 2009-11-06 17:00:00. Period is 60 minutes (1.00 hours)
Uptime: 100.0% (Up: 1 days, 05:34:00 hours)

...
...
...

Replication:
    Master status file number: 541, position: 825383347
    Relay log space limit: 10737418240, used: 4512632192  (42.0%)
    Seconds behind master: 47185
    Estimated time for slave to catch up: 142984 seconds (1 days, 15:43:05 hours)  ETA: 2009-11-08 07:43:05

The human report suggested it would take more than a day and a half for replication to catch up; it also provided with ETA: '2009-11-08 07:43:05'.

As an interesting note, to issue the above report I went back in time and issued:

SELECT report FROM sv_report_human_hour WHERE ts='2009-11-06 16:00:00'\G

It's possible to retroactively generate report, for as long as data still exists.

Checking up on relay log space

Slaves are limited in relay log space. They can only accumulate as much relay logs.

Usually the slave SQL thread is slower than the IO thread, which means when the slave lags behind, it writes more and more relay logs, while having hard time reading them. Once it runs out of relay log space, it stops asking the master for more binary logs, and waits for relay log space to free.

How much relay log space should a slave have?

Well, if you're not worries about disk space, let it have as much as it would like to; but let's look at the implications of relay log space limit. Again, by either looking at the HTML report or by directly issuing:

mysql> SELECT relay_log_used_mb FROM sv_report_chart_hour\G
*************************** 1. row ***************************
relay_log_used_mb: http://chart.apis.google.com/chart?cht=lc&chs=400x200&chts=303030,12&chtt=Nov+5,+10:00++-++Nov+10,+09:00+(4+days,+23+hours)&chdl=relay_log_space_limit_mb|relay_log_space_mb&chdlp=b&chco=ff8c00,4682b4&chd=s:999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999,9999999999989570xytsuxursojfhcabWRSOJEGBCBCDEGBCEBDEFGBEGBDFBCEFGBCDEGBDFBDFCFACEGBCDEGBCBBDEFACEFBCEGBCDEFABCEBEDFACDEF&chxt=x,y&chxr=1,0,10240.10309792&chxl=0:||Nov+6,+09:00|Nov+7,+09:00|Nov+8,+09:00|Nov+9,+09:00|&chxs=0,505050,10

We get:

I see that 2 days lag (comparing with the above charts) will cause the relay log space to fill up. I would consider this to be OK, since I don't normally expect such lags. I can see that a one-day lag is well within relay log space. I conclude that the relay_log_space_limit server variable is properly configured in my case.

http://chart.apis.google.com/chart?cht=lc&chs=400x200&chts=303030,12&chtt=Nov+5,+10:00++-++Nov+10,+08:00+(4+days,+22+hours)&chdl=seconds_behind_master_psec&chdlp=b&chco=ff8c00&chd=s:fbdebXU0wvOgaYXVUIutFTacefXXS4xhYRTJAKf90444444444444444444444482444444445444444444444472544444444444444444444481544444&chxt=x,y&chxr=1,-2.35,0.19&chxl=0:||Nov+6,+09:00|Nov+7,+09:00|Nov+8,+08:00|Nov+9,+08:00|&chxs=0,505050,10