One comment is by Peter Laursen, who rejected the generalization in my post, and another by wlad, who was harsher (but to the point), suggesting my post translated to "I don't know it, therefore it sucks".

I must have delivered the wrong message, since apparently people read my post as "don't use visual editors, they are bad for you", which is not what I intended to say, nor is it what I actually think. I took a very specific aspect of visual editors and commented on that. My comment should not extrapolate to "anything about visual editors is bad".

I don't think people should only be using the command line, and am not in the least interested in vi/emacs/eclipse/netbeans kind of wars. Of course everyone should be using whatever works best for them in terms of productivity and ease of use. And of course visual editors have great advantages over command line.

I'm sorry, it would be childish to assume otherwise, and extra-extra-extrapolation to suggest that was my hidden meaning.

Reflections on Peter's comments

I can see that my post was harsh on visual editors, and agree one cannot just generalize on the dozens of editors available. I have used a few tools, and only to some extent, since I personally feel more comfortable with command line, and so I cannot possibly generalize on all tools. Nor are they all equal.

My point is, there is one de-facto tool, which is the command-line, for which there are no hidden assumptions. For which I don't need to investigate whether a new connection is being opened, or if my query is somehow modified, or... I guess one can say "hey, my tool does nothing of the sort and acts just like the command line". I'm not arguing. The visual editor is an additional layer between you and the server (the mysql command line client is in itself a layer, but a very transparent one, a very low level one), one which needs to take care of visual issues, such as display space, GUI components, widgets not running out of memory, responsiveness, good looks. These, one way or the other, add complexity to the discussion between yourself and your database.

One can say "it's all configurable, there's good documentation, you should know the tool you are using". To some extent the same can be said of the command line client. Here is an observation of mine, please consider it as user input: many (I don't know if most) users will not bother modifying the settings. Many will feel uncomfortable with turning checkboxes on and off, or modify thresholds. It is similar in essence to users not modifying the default InnoDB settings for the MySQL server, working with 8MB of buffer pool.

Anyway I agree my post was over-generalizing on the criticism of visual editors. Sorry for that. It should have been more specific on the advantages of command line.

Reflections on wlad's comments

Since I wanted to concentrate on the command line issue, I didn't elaborate on other stuff. For example, I didn't specify on what type of trainings I ask students to close down their visual editors and open up a command line. Perhaps one could read as "from this moment on, and for the entire duration of the training, no one should be using visual editors, and anyone who does will find himself/herself out of the class". I really didn't think to elaborate on that, and even now - wow, it will take me ages to tell the full story. So I will be extra-brief:

I do make that suggestion. I do that when I start discussion of keys & execution plans. I don't enforce that. I don't mind which tool a developer uses. I actually tell them about various GUI tools they can use (naming the ones I see as most popular among other students).

But I also do not turn my training to "I'm going to do this session using the same tools you are using just because you are using them and you are in this class".

There is a multi-layered reasoning for that:

• Many times there is a mixture of tools in the classroom. I can't just pick up on one. The command line is the most common tool everyone can agree upon. I sometimes demonstrate on MySQL Workbench since many are familiar with it being an "official" GUI tool by the same company which makes the database.
• If I dwell on the particulars of a GUI tool, this consumes precious time that must be used to explain the EXPLAIN and talk about INDEX in depth. I just don't have that time on training. The command line consumes very little time for me.
• I also don't see that I should demonstrate using commercial products. I have nothing against commercial, and on occasion I buy commercial, and I certainly do not slight a commercial product just because it's commercial, but I prefer to sponsor open source. If students have a commercial product, I don't see this is my duty to walk them through the product.
• There are actually a few more reasons; operating systems, installations, assumptions on installed software, strict system administrators who would not installed anything I would tell them, projectors, font sizes, contrast of colors on projected image, ... Believe me, I've been doing this for years, there are many factors to any small decision in class.

Last, I wish to relate to wlad's "I don't know it. It sucks" attribution to my post.

It is true there is not one single SQL GUI tool I know bottoms up. And the status is deteriorating, since I've long since decided my personal preference is command line. This makes me a non-expert, no-authority on any GUI tool. I actually present myself as such in class. This does not lower the quality of the training; it's an aspect outside the training (for the reasons I mentioned above). I therefore agree to "I don't know it".

Now, I strongly believe I do not make the connection to "therefore it sucks". I think that's an exaggerated extrapolation of what I said. But it is such an overwhelming statement about myself for me to read, that I will keep it in mind. It's a good thing to keep in mind, not only on GUI tools, but also in real life.

I see MySQL Workbench, SQLyog, Toad for MySQL, or several more.

I always humbly suggest they close down their software and open up a command line.

It isn't fancy. It may not even be convenient (especially on Windows, in my opinion). And repeating your last command with a minor modification requires a lot of key stroking. Or you would copy+paste from some text editor. Most students will give it a shot, then go back to their favorite editor.

Well, again and again I reach the same conclusion:

Visual editors are not as trustworthy as the command line.

Time and again students show me something on their editor. Behavior seems strange to me. Opening up a console shows a completely different picture.

Things like:

• The visual editor would open a new connection for every new query (oh, so the @user_defined_variable I've just assigned turns NULL, or the TEMPORARY TABLE disappears).
• The visual editor will only show 1,000 results, via LIMIT 0,1000. "But the same query runs so much faster on my machine!". Well, sure, a filesort of 1,000,000 rows that can satisfy the first 1,000 will quit early!
• The visual editor shows table definition graphically. "I didn't realize the index did(n't) cover this and that columns. I didn't realize it only covered first n characters of my VARCHAR.". That's because you can't beat SHOW CREATE TABLE, the definite table structure description.
• The visual editor allows for export/import/copy/transfer of tables and rows with just one click! "Why is it so complicated in the command line to purge 1,000,000 rows from a table?". Ummm, did you realize the visual editor would typically use a naive approach of doing everything in one huge transaction?
• The visual editor is smart. But sometimes you don't want smart. You just assume simple. I personally take great precaution with smart solutions. Luckily, with scripts you have so much greater control (i.e. command line options, "dry-run" mode, etc.) that I have greater confidence in them.

I do like it when a visual editor plays it both smart and safe, in such way that before doing its smart work it actually presents you with the query it's going to issue. Which is why I always considered MySQL Query Browser (now replaced by Workbench) to be the visual editor of choice in my classes.

But, at the end of the day, I strongly believe: if you don't know how to do it with command line, you can't really know how it's done.

We usually test statements using a WHERE clause:

SELECT * FROM world.City WHERE Population > 1000000

The "Population > 1000000" statement makes for a boolean expression. Using WHERE is just one way of evaluating it. One can also test with IF():

SET @val := 7;
SELECT IF(@val > 2, 'Yes', 'No')

TRUE and FALSE

The two are keywords. They also map for the numerals 1 and 0, as follows:

mysql> SELECT TRUE, FALSE;
+------+-------+
| TRUE | FALSE |
+------+-------+
|    1 |     0 |
+------+-------+

Like in the C programming language, a nonzero value evaluates to a true value. A zero evaluates to false. A NULL evaluates to... well, NULL. But aside from 3-valued logic, what's important in our case is that it is not true.

However, simple value comparison is incorrect:

mysql> SELECT @val, @val > 3, @val > 3 = TRUE as result;
+------+----------+--------+
| @val | @val > 3 | result |
+------+----------+--------+
|    7 |        1 |      1 |
+------+----------+--------+

mysql> SELECT @val, @val = TRUE as result;
+------+--------+
| @val | result |
+------+--------+
|    7 |      0 |
+------+--------+

To test for the truth value of an expression, the correct syntax is by using IS TRUE:

SELECT @val, @val IS TRUE as result;
+------+--------+
| @val | result |
+------+--------+
|    7 |      1 |
+------+--------+

Likewise, one may use IS FALSE to test for falsehood. However, if you wish to note NULL as a false value this does not work:

SELECT @empty, @empty IS TRUE, @empty IS FALSE;
+--------+----------------+-----------------+
| @empty | @empty IS TRUE | @empty IS FALSE |
+--------+----------------+-----------------+
| NULL   |              0 |               0 |
+--------+----------------+-----------------+

If you're unsure why, you should read more on three-valued logic in SQL. To solve the above, simply use IS NOT TRUE:

SELECT @empty, @empty IS NOT TRUE;
+--------+--------------------+
| @empty | @empty IS NOT TRUE |
+--------+--------------------+
| NULL   |                  1 |
+--------+--------------------+

In summary, use IS TRUE and IS NOT TRUE so as to normalize truth values into a 0, 1 value range, C style, including handling of NULLs.

The matter of keeping the documentation faithful is a topic of interest. I'd like to outline a few documentation bundling possibilities, and the present the coming new documentation method for common_schema. I'll talk about any bundling that is NOT man pages.

High level: web docs

This is the initial method of documentation I used for openark kit and mycheckpoint. It's still valid for mycheckpoint. Documentation is web-based. You need Internet access to read it. It's in HTML format.

Well, not exactly HTML format: I wrote it in WordPress. Yes, it's HTML, but there's a lot of noise around (theme, menus, etc.) which is not strictly part of the documentation.

While this is perhaps the easiest way to go, here's a few drawbacks:

• You're bound to some framework (WordPress in this case)
• Docs are split between MySQL database (my underlying WordPRess storage) & WordPress files (themes, style, header, footer etc.)
• Documentation is separate from your code - they're just not in the same place
• There is no version control over the documentation.

The result is a single source of documentation, which applies to whatever version is latest. It's impossible to maintain docs for multiple versions. You must manually synchronize your WordPress updates with code commits (or rather - code release!).

Mid level: version controlled HTML docs

I first saw this approach on Baron's Aspersa gets a user manual post. I loved it: the documentation is HTML, but stored as part of your project's code, in same version control.

This means one can browse the documentation (openark kit in this example) exactly as it appears in the baseline. Depending on your project hosting, one may be able to do so per version.

The approach has the great benefit of having the docs tightly coupled with the code in terms of development. Before committing code, one updates documentation for that code, then commits/releases both together.

You're also not bound to any development framework. You may edit with vim, emacs, gedit, bluefish, eclipse, ... any tool of your choice. It's all down to plain old text files.

Mid level #2: documentation bundling

One thing I started doing with common_schema is to release a doc bundle with the code. So one can download a compressed bundle of all HTML files. That way one is absolutely certain what's the right documentation for revision 178. There's no effort about it: the docs are already tightly coupled with code versions. Just compress and distribute.

Low level: documentation coupled with your code

Perl scripts can be written as Perl modules, in which case they are eligible for using the perldoc convention. You code your documentation within your script itself, as comment. Perldoc can extract the documentation and present in man-like format. Same happens with Python's pydoc. Baron's When documentation is code illustrates that approach. Maatkit (now Percona Toolkit) has been using it for years.

This method has the advantage of having the documentation ready right within your shell. You don't need a browser, nor firewall access. The docs are just there for you in the same environment where you're executing the code.

SQL Low level: CALL for help()

common_schema is a different type of project. It is merely a schema. There's no Perl nor Python. One imports the schema into one's MySQL server.

What's the low-level approach for this type of code?

For common_schema I use three levels of documentation: the mid-level, where one can browse through the versioned docs, the 2nd mid-level, where one can download bundled documentation, and then a low-level approach: documentation embedded within the code.

MySQL's documentation is also built into the server: see the help_* tables within the mysql schema. The mysql command line client allows one to access help by supporting the help command, e.g.

mysql> help create table;

The client intercepts this command (this is not server side command) and searches through the mysql.help_* docs.

With common_schema, I don't have control over the client; it's all on server side. But the code being a schema, what with stored routines and tables, it's easy enough to set up documentation.

As of the next version of common_schema, and following MySQL's method, common_schema provides a help table:

DESC help;
+--------------+-------------+------+-----+---------+-------+
| Field        | Type        | Null | Key | Default | Extra |
+--------------+-------------+------+-----+---------+-------+
| topic        | varchar(32) | NO   | PRI | NULL    |       |
| help_message | text        | NO   |     | NULL    |       |
+--------------+-------------+------+-----+---------+-------+

And a help() procedure, so that you can call for help(). The procedure will look for the best matching document based on your search expression:

root@mysql-5.1.51> CALL help('match');
+-------------------------------------------------------------------------------+
| help                                                                          |
+-------------------------------------------------------------------------------+
|                                                                               |
| NAME                                                                          |
|                                                                               |
| match_grantee(): Match an existing account based on user+host.                |
|                                                                               |
| TYPE                                                                          |
|                                                                               |
| Function                                                                      |
|                                                                               |
| DESCRIPTION                                                                   |
|                                                                               |
| MySQL does not provide with identification of logged in accounts. It only     |
| provides with user + host:port combination within processlist. Alas, these do |
| not directly map to accounts, as MySQL lists the host:port from which the     |
| connection is made, but not the (possibly wildcard) user or host.             |
| This function matches a user+host combination against the known accounts,     |
| using the same matching method as the MySQL server, to detect the account     |
| which MySQL identifies as the one matching. It is similar in essence to       |
| CURRENT_USER(), only it works for all sessions, not just for the current      |
| session.                                                                      |
|                                                                               |
| SYNOPSIS                                                                      |
|                                                                               |
|                                                                               |
|                                                                               |
|        match_grantee(connection_user char(16) CHARSET utf8,                   |
|        connection_host char(70) CHARSET utf8)                                 |
|          RETURNS VARCHAR(100) CHARSET utf8                                    |
|                                                                               |
|                                                                               |
| Input:                                                                        |
|                                                                               |
| * connection_user: user login (e.g. as specified by PROCESSLIST)              |
| * connection_host: login host. May optionally specify port number (e.g.       |
|   webhost:12345), which is discarded by the function. This is to support      |
|   immediate input from as specified by PROCESSLIST.                           |
|                                                                               |
|                                                                               |
| EXAMPLES                                                                      |
|                                                                               |
| Find an account matching the given use+host combination:                      |
|                                                                               |
|                                                                               |
|        mysql> SELECT match_grantee('apps', '192.128.0.1:12345') AS            |
|        grantee;                                                               |
|        +------------+                                                         |
|        | grantee    |                                                         |
|        +------------+                                                         |
|        | 'apps'@'%' |                                                         |
|        +------------+                                                         |
|                                                                               |
|                                                                               |
|                                                                               |
| ENVIRONMENT                                                                   |
|                                                                               |
| MySQL 5.1 or newer                                                            |
|                                                                               |
| SEE ALSO                                                                      |
|                                                                               |
| processlist_grantees                                                          |
|                                                                               |
| AUTHOR                                                                        |
|                                                                               |
| Shlomi Noach                                                                  |
|                                                                               |
+-------------------------------------------------------------------------------+

I like HTML for documentation. I think it's a good format, provided you don't start doing funny things. Perhaps TROFF is more suitable; certainly more popular on Unix machines. But I already have everything in HTML. So, what do I do?

My decision was to keep documentation in HTML, and use the handy html2text tool to do the job. And it does it pretty well! The sample you see above is an automated translation of HTML to plain text.

I add a few touches of my own: SELECTing long texts is ugly, whether you do it via ";" or "\G". The help() routine breaks the text by '\n', returning a multi row result set. The above sample makes for some 60+ rows, nicely formatted, broken from the original single text appearing in the help table.

So now you have an internal help method for common_schema, right where the code is. You don't have to leave the command line client in order to get help.

Giuseppe offered me the idea for this, even while my own thinking about it was in early stages.

The next version of common_schema will be available in a few weeks. The code is pretty much ready. I just need to work on, ahem..., the documentation.

Technologist of the Year Award is given for individuals for their technical achievements with regard to Oracle products.

As opposed to community based awards, to win this award one must be nominated by himself or his company. There are several award categories: Developer, DBA, IT Manager etc., and many nominations per category. I have been nominated by my company and am happy to have won the award in the Developer category.

Allow me to take the pleasant opportunity to make some acknowledgements.

makam

Most people know me as a MySQL consultant/instructor/speaker/something. I wear several hats; in one I am co-founder and CTO of SFNK, an Israeli company providing the makam system, a specialized service providing valuable insight on our customers increasing need: the need to know what people say about them on the internet. Our product is running for 7 years now, is serving large amounts of data, and is operating several MySQL server nodes.

I've been working with MySQL for over 11 years, but it is throughout my work in makam that I have come to look deeply into query & server optimization, backups, scale out, design etc. We had some demanding features that pushed me into developing scripts and solutions for MySQL, released as open source. Throughout my experience as a DBA (I'm originally a developer) I came to appreciate the MySQL product, on one had, and find usability fallacies, on the other. The fact that our product is DB centric put me in the position to learn as much as I could about SQL.

I thank makam for this. This product has been the test bed for many of my experiments. Thankfully I never erased the database!

The MySQL community

Though I just recently wrote this, I'm happy to acknowledge it again.

Since my work is based off open source products, open source projects, free advice, professional blogs, I have gained much from the open source world in general, and from the MySQL community in particular, that made me want to share some back.

Its wonderful to have people read and comment on my blog posts and ideas; download, use and provide feedback on code I write; attend an occasional talk. All this pushes me in my work. I honestly hope to have provided with valuable content, code or assistance, to just anyone. Anyone and someone who would benefit from the richness of open source and knowledge sharing.

I am humbled by the advantage I got from the community. I have found names behind the community's first anonymous appearance. I've found people who are easy to reach, happy to respond, generous in their advice, experts in their field.

Thank you.

Oracle & MySQL

An observation on this award is that it is given to a MySQL technologist. I have never worked with Oracle (as in the RDBMS) technologies, and my entire nomination is based on my experience with MySQL.

I see significance in this, and believe it reaffirms the focus and attention MySQL gets from Oracle. It makes me happy as a MySQL user and as a MySQL community member.

I thank Oracle for giving me this honor.

Some things have been done before, done pretty well (if I may say so), and for free.

mycheckpoint is a free monitoring tool for MySQL. It uses an unorthodox approach of monitoring your MySQL servers by storing the monitored data in SQL format (actually, within a MySQL server). This allows you to either use the fancy JavaScript charting (by running mycheckpoint as HTTP server), or Do-It-Yourself diagnostics.

It allows for a great many more things, among which are custom queries and charts: you may add your own queries, to be included in the monitoring scheme. Query your AUTO_INCREMENT values to detect growth rate, query your INFORMATION_SCHEMA tables (if you dare) to learn about your data dimensions, query for query response time to detect performance issues with your website/database, or query your data itself (e.g. sales per day) to get insight about your product. Here are sample charts to illustrate.

Adding a custom query is as easy as producing an INSERT command. And of course you can set up alert conditions. You can set them up on your normal monitored data (e.g. seconds behind master) or on your custom monitored data (e.g. time it takes to execute cart-items query is too long, rate of purchases is below some threshold etc.). It works the same since all monitored data, standard & custom, is stored in SQL format: rows, columns, a table.

This also means you can retroactively check for different thresholds; you can query your data anyway you like! mycheckpoint provides you with a good visualization tool, but, like a good command like client, leaves the data intact for you to analyze the way you like it.

Of course you can get emailed once an alert pops up. But wait, not so fast: you can ask the alert to pop up only if the threshold is trespassed for a given amount of time. For example, it may be OK for OS load average to increase for a few moments, but you may wish to be alerted if this happens for over 30 minutes continuously.

The thing is, you get all these in an uncompressed 222K single python script (43K compressed), and there's no trial period. mycheckpoint is completely free to download and use under the BSD license. It is listed in 10 essential MySQL tools for admin, and is well documented.

I'm not suggesting the scope of mycheckpoint levels with that of other products. It is lightweight; it does not have the full featured web interface as others do; there's work to be done. It only provides basic OS monitoring, and there are missing parts, like replication topology monitoring. But it is feature rich. As the author of mycheckpoint I take notice of using it on production for my own servers. It gets the job done.

It uses some ideas based on existing tools, and a few original ones. Before you say "like never before", take a look at mycheckpoint.

INFORMATION_SCHEMA Optimizations are meant to make your INFORMATION_SCHEMA queries lighter and safer.

For example, if you're going to query the COLUMNS table for just the columns of a single table, then the following:

SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA='sakila' AND TABLE_NAME='rental'

makes for an optimization: specifying a literal on TABLE_SCHEMA avoid scanning the directories of other schemata. Specifying a literal on TABLE_NAME avoids checking up on other tables. So it's a one-schema-one-table read operation, as opposed to "first read every single column from all and any single schema and table, then return only those I'm interested in".

Here's the execution plan for the above query:

*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: COLUMNS
         type: ALL
possible_keys: NULL
          key: TABLE_SCHEMA,TABLE_NAME
      key_len: NULL
          ref: NULL
         rows: NULL
        Extra: Using where; Open_frm_only; Scanned 0 databases

What I tried to do is to read the entire COLUMNS table, one schema at a time, one table at a time. I'm good with this taking longer time.

I have a production system on which reads from COLUMNS consistently crash the servers. Well, one read at a time can't do harm, right?

Unfortunately, as the title of this posts reveals, even sequential read of COLUMNS using INFORMATION_SCHEMA optimization does not help: a minute into the process and the client lost connection. The server crashed.

I was expecting that table locks would be released, buffers released etc. One at a time, there wouldn't be a congestion of locks, reads, table cache suffocation etc.

Was actually having high hopes for this to succeed. I have to find a way in which INFORMATION_SCHEMA tables are not dangerous.

A few hours later, and I have both conclusions and achievements.

There are indeed memory issues with querying from INFORMATION_SCHEMA tables. I've found that VARCHAR(64) columns can consume 64K each: I'm reading from large tables of more than 1,000 columns each, while monitoring MySQL's memory consumption. By dividing the increase in memory by the number of rows resulting from a query I sent, and which was for one single columns, I got an almost exact 64K value per row.

So a query on INFORMATION_SCHEMA consumes much more memory than it should. The good news is that this memory is released once the query terminates. So there is no leak into the session memory.

This is combined with a mistake of mine in the way I iterated the tables, such that the problem was amplified: I happened to query much more than I needed, and so got my query's memory bloated. That is to say, I used the INFORMATION_SCHEMA optimizations only partly right, and so got only part of the savings it could offer me.

With better pinpointing I'm now actually able to read from COLUMNS, without crashing my servers, consistently.

I will further look into the 64K issue. That in itself still drains a lot of memory: on my mycheckpoint schema tables a singe table read means > 64MB of query memory down the drain.

In this part I present DBA's handy syntax for schema and table operations and maintenance.

Confession: while I love INFORMATION_SCHEMA's power, I just hate writing queries against it. It's just so much typing! Just getting the list of tables in a schema makes for this heavy duty query:

SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA='sakila' AND TABLE_TYPE='BASE TABLE';

When a join is involved this really becomes a nightmare. I think it's cumbersome, and as result, many do not remember the names and meaning of columns, making for "oh, I need to read the manual all over again just to get that query right". Anyway, that's my opinion.

A SHOW TABLES statement is easier to type, but cannot be integrated into a SELECT query (though we have a partial solution for that, too), and besides, when filtering out the views, the SHOW statement becomes almost as cumbersome as the one on INFORMATION_SCHEMA.

Which is why foreach() offers handy shortcuts to common iterations on schemata and tables, as follows:

Use case: iterate all databases

call foreach('schema', 'CREATE TABLE ${schema}.event(event_id INT, msg VARCHAR(128))');

In the above we execute a query on each database. Hmmm, maybe not such a good idea to perform this operation on all databases? Let's filter them:

Use case: iterate databases by name match

call foreach('schema like wordpress_%', 'ALTER TABLE ${schema}.wp_posts MODIFY COLUMN comment_author VARCHAR(96) NOT NULL');

The above will only iterate my WordPress databases (I have several of these), performing an ALTER on wp_posts for each of those databases.

I don't have to quote the like expression, but I can, if I wish to.

I can also use a regular expression match:

call foreach('schema ~ /^wordpress_[0-9]+$/', 'ALTER TABLE ${schema}.wp_posts MODIFY COLUMN comment_author VARCHAR(96) NOT NULL');

Use case: iterate tables in a specific schema

Time to upgrade our sakila tables to InnoDB's compressed format. We use $(), a synonym for foreach().

call $('table in sakila', 'ALTER TABLE ${schema}.${table} ENGINE=InnoDB ROW_FORMAT=COMPRESSED');

The above will iterate on tables in sakila. I say tables, since it will avoid iterating views (there is still no specific syntax for views iteration). This is done on purpose, as my experience shows there is very little in common between tables and views when it comes to maintenance and operations.

Use case: iterate tables by name match

Here's a interesting scenario: you wish to work on all tables matching some name. The naive approach would be to:

SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 'wp_posts' AND TABLE_TYPE = 'BASE TABLE'

Wait! Are you aware this may bring your server down? This query will open all databases at once, opening all .frm files (though thankfully not data files, since we only check for name and type).

Here's a better approach:

call foreach('table like wp_posts', 'ALTER TABLE ${schema}.${table} ENGINE=InnoDB');

(There's now FULLTEXT to InnoDB, so the above can make sense in the near future!)

The good part is that foreach() will look for matching tables one database at a time. It will iterate the list of database, then look for matching tables per database, thereby optimizing the query on INFORMATION_SCHEMA.

Here, too, I can use regular expressions:

call $('table ~ /^wp_.*$/', 'ALTER TABLE ${schema}.${table} ENGINE=InnoDB');

Conclusion

This is work in the making, but, as someone who maintains a few productions servers, I've already put it to work.

I'm hoping the syntax is easy to comprehend. I know that since I developed it it must be far more intuitive to myself than to others. I've tried to keep close on common syntax and concepts from various programming languages.

I would like to get as much feedback as possible. I have further ideas and thoughts on the direction common_schema is taking, but wish take it in small steps. Your feedback is appreciated!

foreach(collection_to_iterate_over, queries_to_execute_per_iteration_step);

To illustrate what it can do, consider:

call foreach('table in sakila', 'ALTER TABLE ${schema}.${table} ENGINE=InnoDB ROW_FORMAT=COMPACT');

call $('schema like shard_%', 'CREATE TABLE ${schema}.messages (id INT)');

call $('2000:2009', 'INSERT IGNORE INTO report (report_year) VALUES (${1})');

$() stands as a synonym to foreach(). I suspect it should look familiar to web programmers.

The idea for foreach() was introduced by Giuseppe Maxia during a correspondence. At first I was skeptic: this isn't jQuery; this is SQL. Why would I want to use foreach()?

Then Giuseppe provided some use cases, and as I started thinking about it, I found more and more cases where such a tool might considerably reduce scripting overhead and avoid requiring SQL-fu skills. In fact, I have been using it myself for the past few weeks

I provide examples which I hope will convince the reader as for the simplicity of using such syntax. Showing off the types of input foreach() accepts (query, table search, schema search, set of constants, single or double numbers range), and the types of queries it can execute (single, multiple, using placeholders).

I stress that this is not a replacement for common queries (i.e. does not come to replace your common SELECT, UPDATE, DELETE), but more for working out administrative tasks. Nevertheless, the last example in this post does provide with an interesting insight on possible "normal" use.

Use case: using values from query

Let's kill all queries running for over 20 seconds:

call foreach('SELECT id FROM INFORMATION_SCHEMA.PROCESSLIST WHERE time > 20', 'KILL QUERY ${1}');

The thing I like most about foreach() is that it's self explanatory. Nevertheless, I note:

• The KILL command is executed for each process running for more than 20 seconds (I did round up corners, since I didn't check for sleeping processes, for simplicity).
• I also use the ${1} placeholder: much like in awk, this will get the first column in the result set. In our case, it is the single column, id.
• I chose to invoke a single query/command per iteration step.

Compare the above with another solution to the same problem, using eval():

call eval('SELECT CONCAT(\'KILL QUERY \',id) FROM INFORMATION_SCHEMA.PROCESSLIST WHERE TIME > 20');

They both get the same thing done. But foreach() is just a bit more friendly to write (and read).

Let's move to a more complicated example.

Use case: using multiple values from a query, invoking multiple commands

Let's kill some queries, as above, but also write down a log entry so that we know what happened:

call foreach(
  'SELECT id, user FROM INFORMATION_SCHEMA.PROCESSLIST WHERE time > 20',
  'KILL QUERY ${1}; INSERT INTO my_log VALUES (\'have just killed query ${1}, executed by ${2}\');')
;

In the above, for each long running process, we:

• Kill the process' query. id being the first column, is referenced by ${1}.
• INSERT to my_log that said process has been killed. We note both id and user using placeholders ${1} and ${2}, respectively.

It's possible to invoke as many queries/commands per iteration step. It is possible to use placeholders ${1} through ${9}, as well as ${NR}, which works as in awk: it is a row-counter, 1-based.

This example can still be written with eval(), but in much uglier form. I can't just first KILL the processes, then log about them, since by the time I want to log, the queries will not be running; the commands must be coupled. This is naturally done with foreach().

Use case: iterating constant values, invoking DDL

The commands invoked by foreach() can take the form of DML (INSERT/UPDATE/...), DDL (CREATE/ALTER/...) or other (KILL/SET/...). The placeholders can be used anywhere within the text.

Take an installation where different schemata have the same exact table structure. We want to refactor a table on all schemata:

call $('{USA, UK, Japan, NZ}', 'ALTER TABLE db_region_${1}.customer ADD COLUMN personal_notes VARCHAR(4096) CHARSET utf8');

The above translates to the following commands:

ALTER TABLE db_region_USA.customer ADD COLUMN personal_notes VARCHAR(4096) CHARSET utf8;
ALTER TABLE db_region_UK.customer ADD COLUMN personal_notes VARCHAR(4096) CHARSET utf8;
ALTER TABLE db_region_Japan.customer ADD COLUMN personal_notes VARCHAR(4096) CHARSET utf8;
ALTER TABLE db_region_NZ.customer ADD COLUMN personal_notes VARCHAR(4096) CHARSET utf8;

In the above, we:

• Provide a list of constant values. These can be strings, numbers, whatever. They are space delimited.
• Use the ${1} place holder. We can also use ${NR}.

Use case: loop through number sequence

We wish to populate a table with values:

call foreach('1970:2038', 'INSERT INTO test.test_dates (dt) VALUES (DATE(\'${1}-01-01\'))');

The above results with:

mysql> SELECT dt FROM test_dates;
+------------+
| dt         |
+------------+
| 1970-01-01 |
| 1971-01-01 |
| 1972-01-01 |
| 1973-01-01 |
| 1974-01-01 |
...
| 2036-01-01 |
| 2037-01-01 |
| 2038-01-01 |
+------------+

With numbers range:

• Integers are assumed
• Range is indicated by low and high values, both inclusive
• Negatives allowed (e.g. '-5:5', resulting with 11 steps)
• Placeholders ${1} and ${NR} are allowed.

Use case: iterating through two dimensional numbers range:

We use 3 template tables; we create 15 schemata; in each we create 3 tables based on the template tables:

call foreach( '1:15,1:3',
  'CREATE DATABASE IF NOT EXISTS db_test_${1}; CREATE TABLE db_test_${1}.tbl_${2} LIKE db_template.my_table_${2};'
);

Notes:

• Each of the number ranges has the same restrictions and properties as listed above (integers, inclusive, ascending)
• We can now use ${1} and ${2} placeholders, noting the first and second numbers range, respectively.
• We may also use ${NR}, which, in this case, will run 1 through 45 (15 times 3).
• We use multiple queries per iteration step.

Use case: overcoming MySQL limitations

MySQL does not support ORDER BY & LIMIT in multi-table UPDATE and DELETE statements (as noted last year). So we cannot:

DELETE FROM t1 USING t1 JOIN t2 ON (...) JOIN t3 ON (..) WHERE x = 7 ORDER BY ts LIMIT 100;

However, we can:

call foreach(
  'SELECT t1.id FROM t1 JOIN t2 ON (...) JOIN t3 ON (..) WHERE x = 7 ORDER BY ts LIMIT 100',
  'DELETE FROM t1 WHERE id = ${1}'
);

Of course, it will do a lot of single row DELETEs. There are further MySQL limitations which complicate things if I want to overcome this. Perhaps at a later blog post.

Acknowledgements

I hit a weird bug which prevented me from releasing this earlier on. Actually it's a duplicate of this bug, which makes it 6 years old. Hurray.

To the rescue came Roland Bouman, who suggested an idea so crazy even I was skeptic: to parse and modify the original query so as to rename column names according to my scheme. And of course he made it happen, along with some additional very useful stuff. It's really a super-ultra-meta-meta-sql-fu magic he does there.

So, thanks, Roland, for joining the ride, and thanks, Giuseppe, for testing and helping out to shape this functionality. It's great fun working with other people on open-source -- a new experience for me.

Continued

In this post I've covered the general-purpose iterations. There are also more specific types of iterations with foreach(). Continued next.

common_schema provides with query scripting, analysis & informational views, and a function library, allowing for easier administration and diagnostics for MySQL. It introduces SQL based tools which simplify otherwise complex shell and client scripts, allowing the DBA to be independent of operating system, installed packages and dependencies.

There's no Perl nor Python, and no dependencies to install. It's just a schema.

Some highlights for the new revision:

• foreach(), aka $(): loop through a collection, execute callback commands per element.
• repeat_exec(): a repeat-until device: execute queries until some condition holds.
• exec_file(): execute files a-la SOURCE, but on server side
• Query analysis: analyze query text, view or routine definitions to detect dependency objects.
• Improvements to views and routines, new routines introduced.

Let's take a closer look:

rpbouman

I'm very happy to have Roland Bouman working on this project. He introduced some sophisticated code without which some functionality could not take place. I'm sure I don't need to introduce his great capabilities; I'll just pass the note that it is very good working with him!

foreach()

Introducing a looping device which can iterate a collection and execute callback commands.

What's a collection? A range of numbers; a set of constants; the result set of a SELECT query; tables in your database and more.

What is a callback? A query or set of queries to invoke on the specific elements in the collection. For example:

call foreach('table in sakila', 'ALTER TABLE ${schema}.${table} ENGINE=InnoDB ROW_FORMAT=COMPRESSED');

I'll publish dedicated posts on foreach(), aka $(), following this post. Official documentation is here.

repeat_exec()

Repeat executing queries in a given interval, until some condition holds.

What kind of condition? You can loop forever, or until a given time has passed, a given number of iteration has passed.

You can iterate until no rows are affected by your commands (your callbacks), or until some dynamic condition holds (a query evaluates to true).

For example: purge rows from a table until no more rows are affected; in interval of 3 second:

call repeat_exec(3, 'DELETE FROM test.event WHERE ts < CURDATE() ORDER BY id LIMIT 1000', 0);

Official documentation is here.

exec_file()

If you need to execute commands from a file, you usually invoke SOURCE:

mysql> SOURCE '/tmp/somefile.sql';

Or you invoke mysql client and redirect its input to read from file:

bash$ mysql some_db < /tmp/somefile.sql

SOURCE is a MySQL client command. The file must reside on your client. Running the mysql client is great, but you need to work it out from outside the server.

call_exec() will let you import a file on server side, from within the server:

call exec_file('/tmp/some_file.sql');

You will need to have the file readable; it is limited to 64K at this moment; it may not use DELIMITER, and it may not include dynamic SQL. These are the limitations.

Official documentation is here.

exec() / exec_single()

All of the above rely on the exec() / exec_single() routines, which dynamically execute a set of queries. One one hand, it's no big deal: they only have to use prepared statements in order to invoke the queries. But then, they knows how to parse multiple queries (find the ";" delimiter correctly), plus they allow for configuration: if you set @common_schema_dryrun, queries are not actually executes; just printed out. If you set @common_schema_verbose, queries are verbosed in addition to being executed. Since all execution routines rely on these,we get a standardized pattern.

Official documentation is here.

Query analysis

Query parsing routines allow for detection of dependencies within queries. While not full-blown SQL parser, these allow one to realize on which tables or routines a view depends on; or a routines depends on; or an event; or just any query.

These routines can analyze the text of not only a SELECT query, but also UPDATE, DELETE, CREATE, etc. They can read the code of a stored routines, including queries and control flow constructs; thus, they are also able to analyze events and triggers.

At this stage forward-dependencies resolution is supported. This can eventually lead to dependency graphs or to reverse-dependency resolution (i.e. "which view, routine, trigger or event depend on table t?")

Example:

mysql> call get_view_dependencies('sakila', 'actor_info');
+-------------+---------------+-------------+--------+
| schema_name | object_name   | object_type | action |
+-------------+---------------+-------------+--------+
| sakila      | actor         | table       | select |
| sakila      | category      | table       | select |
| sakila      | film          | table       | select |
| sakila      | film_actor    | table       | select |
| sakila      | film_category | table       | select |
+-------------+---------------+-------------+--------+

The query analysis routines are in BETA stage.

Official documentation is here.

Test quite

common_schema is now tested. Not all code is as yet under tests; all new code is, and some of the older code. Work is in progress to add more and more tests.

Further changes:

• candidate_keys does not give higher score for PRIMARY KEYs any longer. It ranks all unique keys according to its own heuristic; it also provides with the  is_primary and is_nullable columns.
• Added candidate_keys_recommended view, recommending best candidate key per table (while noting whether it qualifies as PRIMARY KEY in terms of NULLable columns).
• Added many text parsing and text manipulation routines, such as better trim, tokenizing, etc. Improved existing code significantly.

Get it

common_schema is available for downloaded. It is released under the BSD license, and is free.

I've put very hard work into common_schema's documentation. It is very thorough and provides with clear examples. The documentation is also available for download.

If you encounter problems, please report on the issues page.

common_schema is meant to be downloaded & installed on any MySQL server. It provides with general and essential functionality. Spread the word!